All authors
Claude Skills by feiyang-k
github.com/feiyang-k258 skills0 installs459 views
- Git A Generative Image To Text Transformer For Vision And Language Arxiv 2205 14100v5Use this skill when you want a simple generative image-to-text model trained on large-scale image-text data with minimal architectural complexity. Avoid it when you need a more complex architecture with cross-attention or dual encoders.Votes: 0GitHub stars: 9
- Glip Grounded Language Image Pre Training Arxiv 2112 03857v2Use this skill when you want to unify object detection and phrase grounding training data for a model that detects objects from language descriptions. Avoid it when you do not need language-guided detection or grounding.Votes: 0GitHub stars: 9
- Gpt 4vision System Card Openai Gpt4v 2023Use this skill when you want to understand safety evaluation and mitigation strategies for deploying multimodal LLMs. Avoid it when safety evaluation is not your focus.Votes: 0GitHub stars: 9
- Grounding Dino Marrying Dino With Grounded Pre Training For Open Set Object Detection Arxiv 2303 05499v4Use this skill when you need an open-set object detector trained on grounded text-image data to detect any object described in natural language. Avoid it when you only need closed-set detection on fixed categories.Votes: 0GitHub stars: 9
- Idefics2 An 8b Parameters Multimodal Model Arxiv 2405 02246v2Use this skill when you want an efficient 8B VLM training recipe with curated data combining web interleaved, paired, and instruction data. Avoid it when you need a larger model or have a different data strategy.Votes: 0GitHub stars: 9
- Imagenet 21k Pretraining For The Masses Arxiv 2104 10972v4Use this skill when you want to use ImageNet-21K for vision encoder pretraining with proper preprocessing and training recipes. Avoid it when ImageNet-1K or CLIP pretraining is sufficient.Votes: 0GitHub stars: 9
- Improved Baselines With Visual Instruction Tuning Arxiv 2310 03744v2Use this skill when you want to curate a high-quality academic-task-oriented instruction dataset for VLM fine-tuning. Avoid it when you need only synthetic conversation data or cannot access the academic VQA datasets.Votes: 0GitHub stars: 9
- Improving Multimodal Learning With Multi Loss Gradient Modulation Arxiv 2405 07930v2Use this skill when you want to optimize multi-task VLM training by dynamically modulating gradients from different loss functions. Avoid it when single-loss training is sufficient.Votes: 0GitHub stars: 9
- Instructblip Towards General Purpose Vision Language Models With Instruction Tuning Arxiv 2305 06500v2Use this skill when you want to curate multi-task instruction data and make the vision encoder instruction-aware for better VLM fine-tuning. Avoid it when you have a single task and do not need instruction-conditioned visual feature extraction.Votes: 0GitHub stars: 9
- Internlm Xcomposer2 Mastering Free Form Text Image Composition And Comprehension Arxiv 2401 16420v2Use this skill when you want training data for a VLM that can both understand and compose interleaved text-image content like articles. Avoid it when you only need image understanding without composition.Votes: 0GitHub stars: 9
- Internvl Scaling Up Vision Foundation Models And Aligning For Generic Visual Linguistic Tasks Arxiv 2312 14238v5Use this skill when you need a progressive data pipeline that aligns a large vision encoder with an LLM through contrastive, generative, and instruction-tuning stages. Avoid it when you are building a small model and cannot afford multi-stage pretraining.Votes: 0GitHub stars: 9
- Internvl2 Better Than The Best Expanding Performance Boundaries Of Open Source Multimodal Models With A Progressive St Arxiv 2404 16821v2Use this skill when you want to progressively scale VLM data from alignment to multi-task to high-quality instruction data with dynamic resolution. Avoid it when you do not have multi-stage data or cannot afford progressive training.Votes: 0GitHub stars: 9
- Label Noise Sgd Provably Prefers Flat Global Minimizers Arxiv 2106 06530v4Use this skill when you want to understand how label noise in training data affects optimization and generalization, particularly for noisy web data. Avoid it when you do not deal with noisy labels.Votes: 0GitHub stars: 9
- Lima Less Is More For Alignment Arxiv 2305 11206v1Use this skill when you want to align an LLM with just 1,000 carefully curated instruction examples, demonstrating that quality trumps quantity. Avoid it when you have abundant alignment data or need maximum diversity.Votes: 0GitHub stars: 9
- Llama 3 The Llama 3 Herd Of Models Arxiv 2407 21783v2Use this skill when you want to understand Meta's comprehensive approach to pretraining data curation covering text, code, multimodal, and multilingual data at massive scale. Avoid it when you do not need insights into large-scale pretraining data recipes.Votes: 0GitHub stars: 9
- Llama Adapter Efficient Fine Tuning Of Language Models With Zero Init Attention Arxiv 2303 16199v2Use this skill when you want to efficiently add visual instruction-following to LLaMA using lightweight adapters with zero-initialized attention. Avoid it when full fine-tuning is feasible or you prefer other PEFT methods.Votes: 0GitHub stars: 9
- Llama Guard Llm Based Input Output Safeguard For Human Ai Conversations Arxiv 2312 06674v2Use this skill when you want to train a safety classifier for filtering harmful content in LLM/VLM inputs and outputs. Avoid it when you do not need safety filtering or have an existing safety classifier.Votes: 0GitHub stars: 9
- Llava Gemma Accelerating Multimodal Foundation Models With A Compact Language Model Arxiv 2404 01331v2Use this skill when you want to adapt the LLaVA data recipe for compact language models like Gemma-2B for efficient VLM deployment. Avoid it when you are using a larger LLM and do not need compact VLMs.Votes: 0GitHub stars: 9
- Llava Next Improved Reasoning Ocr And World Knowledge Arxiv 2401 06209v1Use this skill when you want to scale up instruction tuning data with higher quality and diversity for improved VLM reasoning. Avoid it when you have limited compute and cannot handle dynamic high-resolution image processing.Votes: 0GitHub stars: 9
- Llava Next Video Scaling Video Understanding To Strong Image Level Performance Arxiv 2410 02713v1Use this skill when you want to transfer strong image understanding from LLaVA-NeXT to video through efficient frame sampling and video-specific data. Avoid it when you are building a video-from-scratch VLM.Votes: 0GitHub stars: 9
- Llava Onevision Easy Visual Task Transfer Arxiv 2408 03326v2Use this skill when you want a unified training recipe for single-image, multi-image, and video understanding with curated data at each stage. Avoid it when you only need single-image understanding.Votes: 0GitHub stars: 9
- Llava Rlhf Aligning Large Multimodal Models With Factually Augmented Rlhf Arxiv 2309 14525v4Use this skill when you want to align a VLM using RLHF with factually augmented reward signals that penalize hallucination. Avoid it when you do not need RLHF-based alignment for your VLM.Votes: 0GitHub stars: 9
- Longva Long Context Transfer From Language To Vision Arxiv 2406 16852v2Use this skill when you want to extend VLM context length for processing many images or long videos by transferring long context from language training. Avoid it when you do not need long-context visual processing.Votes: 0GitHub stars: 9
- Lora Low Rank Adaptation Of Large Language Models Arxiv 2106 09685v2Use this skill when you want to efficiently fine-tune LLMs/VLMs on domain-specific data using low-rank adapter matrices instead of full fine-tuning. Avoid it when you can afford full fine-tuning or need maximum model capacity.Votes: 0GitHub stars: 9
- Masked Autoencoders Are Scalable Vision Learners Arxiv 2111 06377v2Use this skill when you want to pre-train a ViT by masking 75% of image patches and reconstructing them, learning rich visual features. Avoid it when contrastive pretraining (CLIP, DINO) is sufficient.Votes: 0GitHub stars: 9
- Matryoshka Representation Learning Arxiv 2205 13147v4Use this skill when you want to train embeddings that work at multiple dimensionalities, enabling flexible compute-quality tradeoffs at retrieval time. Avoid it when you only need fixed-dimension embeddings.Votes: 0GitHub stars: 9
- Minicpm V A Gpt 4v Level Mllm On Your Phone Arxiv 2408 01800v3Use this skill when you want a data recipe for training a phone-deployable VLM that approaches GPT-4V quality on key benchmarks. Avoid it when you are not targeting mobile deployment.Votes: 0GitHub stars: 9
- Mm1 Methods Analysis And Insights From Multimodal Llm Pre Training Arxiv 2403 09611v3Use this skill when you want to understand optimal data mixing recipes for multimodal LLM pretraining across interleaved, image-text, and text-only data. Avoid it when you only have one data type or cannot run ablation studies to tune the mix.Votes: 0GitHub stars: 9
- Monkey Image Resolution And Text Label Are Important Things For Large Multi Modal Models Arxiv 2311 06607v3Use this skill when you want to create instruction data for high-resolution VLMs that can handle documents, dense text, and detailed images. Avoid it when you only work with low-resolution images or do not need document understanding.Votes: 0GitHub stars: 9
- Mplug Owl Modularization Empowers Large Language Models With Multimodality Arxiv 2304 14178v2Use this skill when you want a modular VLM training approach using visual abstractor and two-stage training on alignment then instruction data. Avoid it when you prefer simpler projection approaches.Votes: 0GitHub stars: 9
- Mplug Owl2 Revolutionizing Multi Modal Large Language Model With Modality Collaboration Arxiv 2311 04257v2Use this skill when you want to improve VLM training through modality-adaptive modules and curated multi-task data that promotes vision-language collaboration. Avoid it when you are using a simple vision-language projection without modality adaptation.Votes: 0GitHub stars: 9
- Multimodal Learning With Transformers A Survey Arxiv 2206 06488v2Use this skill when you need a comprehensive survey of multimodal learning with transformers covering fusion strategies, pretraining, and applications. Avoid it when you need a specific method rather than a broad overview.Votes: 0GitHub stars: 9
- Multimodal Neurons In Artificial Neural Networks Arxiv 2103 01002v1Use this skill when you want to understand how CLIP learns multimodal neurons that respond to both visual and textual concepts for debugging data quality. Avoid it when you do not need interpretability or representation analysis.Votes: 0GitHub stars: 9
- Open Vocabulary Object Detection Using Captions Arxiv 2011 10678v2Use this skill when you want to train an object detector using image captions as weak supervision instead of bounding box annotations. Avoid it when you have abundant bounding box annotations.Votes: 0GitHub stars: 9
- Openclip An Open Source Implementation Of Clip Arxiv 2212 07143v2Use this skill when you want to train CLIP models on open datasets (LAION) with reproducible training recipes and systematic data ablations. Avoid it when you are using pre-trained CLIP models and do not need to train your own.Votes: 0GitHub stars: 9
- Pali 3 Smaller Faster Stronger Arxiv 2310 09199v2Use this skill when you want insights on training a smaller but stronger VLM through improved data quality, SigLIP encoder, and efficient training recipe. Avoid it when you are not optimizing VLM training efficiency.Votes: 0GitHub stars: 9
- Pixtral 12b A Frontier Multimodal Model Arxiv Pixtral 2024Use this skill when you want to train a natively multimodal model that processes variable numbers of images at any resolution without fixed token budgets. Avoid it when fixed-resolution VLM approaches work for your needs.Votes: 0GitHub stars: 9
- Pope Polling Based Object Probing Evaluation For Object Hallucination Arxiv 2305 10355v2Use this skill when you want to evaluate and reduce object hallucination in VLMs using a polling-based evaluation framework. Avoid it when you are not concerned about object hallucination or have other evaluation methods.Votes: 0GitHub stars: 9
- Qwen Vl A Frontier Large Vision Language Model With Versatile Abilities Arxiv Qwen2vl 2024Use this skill when you want to understand multi-resolution pretraining strategies where images are processed at progressively higher resolutions during training. Avoid it when single-resolution training is sufficient.Votes: 0GitHub stars: 9
- Qwen Vl A Versatile Vision Language Model For Understanding Localization Text Reading And Beyond Arxiv 2308 12966v3Use this skill when you need a multi-stage data pipeline covering pretraining, multi-task training, and instruction tuning with grounding and OCR data. Avoid it when you need only basic image understanding without grounding or OCR capabilities.Votes: 0GitHub stars: 9
- Qwen2 Vl Enhancing Vision Language Models Perception Of The World At Any Resolution Arxiv 2409 12191v2Use this skill when you want to train a VLM with native dynamic resolution support and multi-stage data covering images, videos, and documents. Avoid it when you use fixed-resolution input and do not need dynamic resolution.Votes: 0GitHub stars: 9
- Recap Datacomp 1b How To Turn A Filtered Dataset Into A Recaptioned One Arxiv 2406 08478v1Use this skill when you want to recaption a billion-scale dataset like DataComp-1B using an open-source VLM to improve CLIP training. Avoid it when your dataset captions are already high-quality or you lack the compute for billion-scale recaptioning.Votes: 0GitHub stars: 9
- Refinedweb A Falcons Recipe For High Quality Web Data Arxiv 2306 01116v1Use this skill when you want to build a high-quality web text corpus through aggressive deduplication and filtering of Common Crawl. Avoid it when you have a curated text corpus or do not need web-scale text data.Votes: 0GitHub stars: 9
- Rlhf V Towards Trustworthy Mllms Via Behavior Alignment From Fine Grained Correctional Human Feedback Arxiv 2312 00849v2Use this skill when you want to collect fine-grained human feedback on VLM outputs to reduce hallucination through RLHF. Avoid it when you do not need RLHF or hallucination is not a concern.Votes: 0GitHub stars: 9
- Scaling Autoregressive Multi Modal Models Pretraining And Instruction Tuning Arxiv 2309 02591v1Use this skill when you need a systematic approach to curating paired image-text and interleaved data at web scale for autoregressive VLM pretraining. Avoid it when you only need a small fine-tuning dataset.Votes: 0GitHub stars: 9
- Scaling Instruction Finetuned Language Models Arxiv 2210 11416v5Use this skill when you want to understand how to scale instruction fine-tuning across number of tasks, model size, and chain-of-thought data for maximum benefit. Avoid it when you are doing single-task fine-tuning.Votes: 0GitHub stars: 9
- Scaling Up Visual And Vision Language Representation Learning With Noisy Text Supervision Arxiv 2111 02114v1Use this skill when you want to improve image-text datasets by replacing web alt-text with richer synthetic captions from a captioning model. Avoid it when you already have high-quality detailed captions or the captioner lacks domain knowledge.Votes: 0GitHub stars: 9
- Scaling Vision Transformers To 22 Billion Parameters Arxiv 2302 05442v1Use this skill when you want to scale vision transformers to 22B parameters using JFT data with efficient training techniques. Avoid it when you do not need a very large vision encoder.Votes: 0GitHub stars: 9
- Scaling Vision With Sparse Mixture Of Experts Arxiv 2106 05974v1Use this skill when you want to scale vision models efficiently using sparse Mixture-of-Experts for handling more training data per compute. Avoid it when dense models are sufficient for your scale.Votes: 0GitHub stars: 9
- Semdedup Data Efficient Learning At Web Scale Through Semantic Deduplication Arxiv 2303 09540v2Use this skill when you want to remove semantically redundant examples from large-scale training data using embedding-based deduplication. Avoid it when you only need exact or near-exact duplicate removal or your dataset is small enough that redundancy is not a concern.Votes: 0GitHub stars: 9