All authors
Claude Skills by feiyang-k
github.com/feiyang-k258 skills0 installs459 views
- Drivelm Driving With Graph Visual Question Answering Arxiv 2312 14150v2Use this skill when you want to create structured QA data for autonomous driving VLMs using graph-based scene understanding. Avoid it when you do not work with autonomous driving or need general-purpose VQA.Votes: 0GitHub stars: 9
- Ferret Refer And Ground Anything Anywhere At Any Granularity Arxiv 2310 07704v2Use this skill when you want to create training data for a VLM that can both refer to and ground objects at point, box, and free-form region granularities. Avoid it when you only need image-level understanding without spatial referring.Votes: 0GitHub stars: 9
- Florence 2 Advancing A Unified Representation For A Variety Of Vision Tasks Arxiv 2311 06242v2Use this skill when you need to build a large-scale multi-task dataset with diverse vision annotations (caption, detection, segmentation, grounding) using automated annotation engines. Avoid it when you only need a single task type or already have comprehensive multi-task annotations.Votes: 0GitHub stars: 9
- Geochat Grounded Large Vision Language Model For Remote Sensing Arxiv 2311 15826v3Use this skill when you want to create instruction data for remote sensing VLMs covering satellite image understanding, change detection, and visual grounding. Avoid it when you do not work with remote sensing or satellite imagery.Votes: 0GitHub stars: 9
- Glamm Pixel Grounding Large Multimodal Model Arxiv 2311 03356v2Use this skill when you want to create training data for a VLM that generates text grounded at the pixel level with segmentation masks. Avoid it when you do not need pixel-level grounding in VLM outputs.Votes: 0GitHub stars: 9
- Gqa A New Dataset For Real World Visual Reasoning And Compositional Question Answering Arxiv 1902 09506v3Use this skill when you want to generate balanced, compositional visual QA data from scene graphs using programmatic question generation. Avoid it when you need free-form questions that cannot be generated from scene graph templates.Votes: 0GitHub stars: 9
- Grounded Sam Assembling Open World Models For Diverse Visual Tasks Github Grounded Sam 2024Use this skill when you want to automatically generate grounded segmentation annotations by combining Grounding DINO for text-based detection with SAM for segmentation. Avoid it when you have manual grounding annotations or do not need automated annotation.Votes: 0GitHub stars: 9
- Hard Negative Mixing For Contrastive Learning Arxiv 2010 01028v2Use this skill when you want to generate harder negative examples for contrastive learning by mixing embeddings of existing negatives. Avoid it when random negatives provide sufficient contrastive signal.Votes: 0GitHub stars: 9
- Imagen Photorealistic Text To Image Diffusion Models With Deep Language Understanding Arxiv 2205 11487v1Use this skill when you want to leverage a frozen large language model as a text encoder for high-fidelity text-to-image generation. Avoid it when CLIP text encoders are sufficient for your generation needs.Votes: 0GitHub stars: 9
- Instruction Tuning With Gpt 4 Arxiv 2304 03277v1Use this skill when you want to generate high-quality instruction data using GPT-4 for fine-tuning smaller models. Avoid it when you cannot afford GPT-4 API calls or have sufficient human-written data.Votes: 0GitHub stars: 9
- Instructiongpt 4 A 200 Instruction Paradigm For Fine Tuning Minigpt 4 Arxiv 2308 12058v2Use this skill when you want to align a VLM with just 200 carefully curated high-quality instruction examples, pushing the LIMA principle to multimodal. Avoid it when you have more than 200 good instruction examples available.Votes: 0GitHub stars: 9
- Instructpix2pix Learning To Follow Image Editing Instructions Arxiv 2211 09800v2Use this skill when you want to generate synthetic image editing instruction data by combining GPT-3 text editing with Prompt-to-Prompt image editing. Avoid it when you have real image editing data or do not need editing capability.Votes: 0GitHub stars: 9
- Kosmos 2 Grounding Multimodal Large Language Models To The World Arxiv 2306 14824v3Use this skill when you need to construct grounded image-text data with bounding box annotations linked to text spans for training grounding-capable VLMs. Avoid it when you do not need spatial grounding or bounding box outputs from your VLM.Votes: 0GitHub stars: 9
- Laclip Improving Clip Training With Language Rewrites Arxiv 2305 20088v2Use this skill when you want to augment CLIP training text by rewriting captions with an LLM to improve text diversity. Avoid it when your captions are already diverse and high-quality or you cannot afford LLM rewriting.Votes: 0GitHub stars: 9
- Layoutlmv3 Pre Training For Document Ai With Unified Text And Image Masking Arxiv 2204 08387v3Use this skill when you want to pre-train a document understanding model using unified text and image masking on document images with OCR. Avoid it when you do not need document layout understanding or OCR-dependent processing.Votes: 0GitHub stars: 9
- Lisa Reasoning Segmentation Via Large Language Model Arxiv 2308 00692v2Use this skill when you want to create training data for a VLM that outputs segmentation masks through language-based reasoning. Avoid it when you do not need segmentation output or can use standard segmentation datasets.Votes: 0GitHub stars: 9
- Llava Critic Learning To Evaluate Multimodal Models Arxiv 2410 02712v1Use this skill when you want to train a VLM to evaluate its own outputs for self-improvement without external AI judges. Avoid it when you have access to external AI judges or do not need self-evaluation.Votes: 0GitHub stars: 9
- Llava Med Training A Large Language And Vision Assistant For Biomedicine In One Day Arxiv 2306 00890v2Use this skill when you want to generate biomedical visual instruction data by aligning PMC image-caption pairs with LLM-generated QA. Avoid it when you need general-purpose instruction data or lack biomedical image-caption sources.Votes: 0GitHub stars: 9
- Llava Video Unified Video Understanding With Video Representation Learning Arxiv 2501 00599v1Use this skill when you want to scale video instruction data for training a unified video understanding VLM with diverse temporal reasoning tasks. Avoid it when you only need image understanding or have limited video data.Votes: 0GitHub stars: 9
- Magpie Alignment Data Synthesis From Scratch By Prompting Aligned Llms With Nothing Arxiv 2406 08464v3Use this skill when you want to generate instruction data by prompting an aligned LLM with just the system prompt to elicit user-like instructions from the model itself. Avoid it when you have instruction data or prefer structured generation approaches.Votes: 0GitHub stars: 9
- Mantis Interleaved Multi Image Instruction Tuning Arxiv 2405 01483v2Use this skill when you want to create instruction data for multi-image understanding tasks like comparison, reasoning across images, and temporal understanding. Avoid it when you only need single-image instruction data.Votes: 0GitHub stars: 9
- Minigpt 4 Enhancing Vision Language Understanding With Advanced Large Language Models Arxiv 2304 10592v2Use this skill when you want to align a VLM using a small set of self-curated high-quality image-description pairs. Avoid it when you have abundant alignment data or do not need to fix generation quality issues.Votes: 0GitHub stars: 9
- Mixup Beyond Empirical Risk Minimization Arxiv 1710 09412v2Use this skill when you want to augment training data by linearly interpolating between pairs of training examples and their labels. Avoid it when you need discrete, unmodified training examples.Votes: 0GitHub stars: 9
- Mosaic Augmentation For Detection And Segmentation Arxiv Yolov4 2020Use this skill when you want to create training images by combining four cropped images into a mosaic grid for object detection training. Avoid it when standard single-image augmentation is sufficient.Votes: 0GitHub stars: 9
- Multimodal Few Shot Learning With Frozen Language Models Arxiv 2106 13884v2Use this skill when you want to enable multimodal few-shot learning by training a visual encoder to produce prefix tokens for a frozen language model. Avoid it when you have abundant training data and do not need few-shot adaptation.Votes: 0GitHub stars: 9
- Nemotron 4 340b Technical Report Arxiv 2406 11704v1Use this skill when you want to build a comprehensive synthetic data generation pipeline using strong models to create instruction data at scale. Avoid it when you have sufficient real instruction data or cannot run large model inference.Votes: 0GitHub stars: 9
- Nougat Neural Optical Understanding For Academic Documents Arxiv 2308 13418v1Use this skill when you want to convert academic PDF documents into structured markup using a vision transformer without traditional OCR. Avoid it when you need OCR for general documents or handwriting rather than academic papers.Votes: 0GitHub stars: 9
- Orca Progressive Learning From Complex Explanation Traces Of Gpt 4 Arxiv 2306 02707v1Use this skill when you want to distill reasoning capability by generating step-by-step explanation traces from a strong teacher model. Avoid it when you do not need reasoning capability or have sufficient reasoning data.Votes: 0GitHub stars: 9
- Osprey Pixel Understanding With Visual Instruction Tuning Arxiv 2312 10032v2Use this skill when you want to create pixel-level instruction tuning data where the VLM can understand and describe individual pixel regions in images. Avoid it when bounding box-level understanding is sufficient.Votes: 0GitHub stars: 9
- Otter A Multi Modal Model With In Context Instruction Tuning Arxiv 2305 03726v2Use this skill when you want to create instruction data that includes in-context examples for multi-modal in-context learning. Avoid it when you do not need in-context learning capability in your VLM.Votes: 0GitHub stars: 9
- Panda 70m Captioning 70m Videos With Multiple Cross Modality Teachers Arxiv 2402 19479v2Use this skill when you want to generate captions for 70M video clips using multiple cross-modality teacher models. Avoid it when you only need image captions or cannot process video at this scale.Votes: 0GitHub stars: 9
- Phi 1 Textbooks Are All You Need Arxiv 2306 11644v2Use this skill when you want to generate synthetic textbook-quality code data for training a small but highly capable code model. Avoid it when you have sufficient high-quality code data or are not training a code model.Votes: 0GitHub stars: 9
- Phi 15 Textbooks Are All You Need Ii Arxiv 2309 05463v2Use this skill when you want to generate synthetic textbook-quality training data to teach reasoning to small language models. Avoid it when you are training a large model where data quality filtering alone suffices.Votes: 0GitHub stars: 9
- Phi 3 Technical Report A Highly Capable Language Model Locally On Your Phone Arxiv 2404 14219v2Use this skill when you want to train high-capability small models using heavily filtered web data augmented with synthetic textbook-quality data. Avoid it when you are training a large model where data quality filtering is less impactful.Votes: 0GitHub stars: 9
- Pix2struct Screenshot Parsing As Pretraining For Visual Language Understanding Arxiv 2210 03347v2Use this skill when you want to pre-train a document understanding model by parsing web page screenshots into their HTML structure. Avoid it when you do not need document understanding or your documents are not web-like.Votes: 0GitHub stars: 9
- Pixart Alpha Fast Training Of Diffusion Transformer For Photorealistic Text To Image Synthesis Arxiv 2310 00426v2Use this skill when you want to train a high-quality text-to-image model efficiently using carefully curated high-quality captions from an LLM. Avoid it when you already have a well-trained text-to-image model.Votes: 0GitHub stars: 9
- Pointllm Empowering Large Language Models To Understand Point Clouds Arxiv 2308 16911v3Use this skill when you want to create instruction data for LLMs to understand 3D point clouds using GPT-4 to generate point cloud descriptions. Avoid it when you do not work with point cloud data.Votes: 0GitHub stars: 9
- Randaugment Practical Automated Data Augmentation With A Reduced Search Space Arxiv 1909 13719v2Use this skill when you want simple automated data augmentation with only two hyperparameters (number and magnitude of transforms). Avoid it when you have a custom augmentation pipeline or do not need automated augmentation.Votes: 0GitHub stars: 9
- Regiongpt Towards Region Understanding Vision Language Model Arxiv 2403 02330v2Use this skill when you want to create region-level instruction data for VLMs that understand and describe image regions with fine-grained detail. Avoid it when image-level understanding is sufficient.Votes: 0GitHub stars: 9
- Segment Everything Everywhere All At Once Arxiv 2304 06718v2Use this skill when you want a unified segmentation model trained on diverse segmentation data types (semantic, instance, panoptic, interactive) for VLM grounding. Avoid it when you only need one segmentation type.Votes: 0GitHub stars: 9
- Self Instruct Aligning Language Models With Self Generated Instructions Arxiv 2212 10560v2Use this skill when you want to bootstrap instruction data from a seed set by having the model generate its own instructions iteratively. Avoid it when you have abundant human-written instructions or need domain expertise the model lacks.Votes: 0GitHub stars: 9
- Sharegpt4v Improving Large Multi Modal Models With Better Captions Arxiv 2311 12793v2Use this skill when you want to generate high-quality detailed captions using GPT-4V and then train a captioner to scale caption generation. Avoid it when you have sufficient high-quality captions or cannot afford GPT-4V API calls for the seed set.Votes: 0GitHub stars: 9
- Sharegpt4video Improving Video Understanding And Generation With Better Captions Arxiv 2406 04325v2Use this skill when you want to generate high-quality video captions using GPT-4V and train an open-source video captioner for scaling. Avoid it when you do not work with video or have sufficient video captions.Votes: 0GitHub stars: 9
- Shikra Unleashing Multimodal Llms Referential Dialogue Magic Arxiv 2306 15195v2Use this skill when you want to create referential dialogue data where the VLM and user exchange spatial references (bounding boxes) naturally within conversation. Avoid it when you do not need spatial references in conversation.Votes: 0GitHub stars: 9
- Silkie Preference Distillation For Large Visual Language Models Arxiv 2312 10665v1Use this skill when you want to generate preference data for VLM alignment using AI feedback instead of human feedback. Avoid it when you can collect human preferences or do not need preference-based alignment.Votes: 0GitHub stars: 9
- Stable Diffusion High Resolution Image Synthesis With Latent Diffusion Models Arxiv 2112 10752v2Use this skill when you want to use text-to-image diffusion models to generate synthetic training images for augmenting VLM training data. Avoid it when you have sufficient real training images or synthetic images would not improve your task.Votes: 0GitHub stars: 9
- Stablerep Synthetic Images From Text To Image Models Make Strong Visual Representation Learners Arxiv 2306 00984v3Use this skill when you want to train CLIP-style models entirely on synthetic images generated by Stable Diffusion instead of real web images. Avoid it when you have sufficient real image data or synthetic quality is insufficient.Votes: 0GitHub stars: 9
- Svit Scaling Up Visual Instruction Tuning Arxiv 2307 04087v2Use this skill when you want to scale up visual instruction tuning data to 4.2M samples using GPT-4 with detailed image annotations. Avoid it when you already have sufficient instruction data or cannot afford GPT-4 generation at scale.Votes: 0GitHub stars: 9
- Synthtext Synthetic Data For Text Detection Arxiv 1604 06646v1Use this skill when you want to generate synthetic images with realistic text overlaid on natural scenes for text detection/recognition training. Avoid it when you have sufficient real text detection data or need non-text image generation.Votes: 0GitHub stars: 9
- Textcraftor Your Text Classifier Is A Prompt Generator For Image Generation Arxiv 2311 01459v2Use this skill when you want to refine text prompts for image generation to produce higher quality, more faithful synthetic training images. Avoid it when your prompts already produce satisfactory synthetic images.Votes: 0GitHub stars: 9