Text-to-LoRA: Instant Transformer Adaption

%PDF-1.5 %???? 1 0 obj << /Metadata 3 0 R /Names 4 0 R /OpenAction 5 0 R /PageMode /UseNone /Pages 6 0 R /Type /Catalog >> endobj 2 0 obj << /Author (Rujikorn Charakorn; Edoardo Cetin; Yujin Tang; Robert Tjarko Lange) /Creator (arXiv GenPDF \(tex2pdf:\)) /DOI (https://doi.org/10.48550/arXiv.2506.06105) /License (http://creativecommons.org/licenses/by-nc-sa/4.0/) /PTEX.Fullbanner (This is pdfTeX, Version 3.141592653-2.6-1.40.25 \(TeX Live 2023\) kpathsea version 6.3.5) /Producer (pikepdf 8.15.1) /Title (Text-to-LoRA: Instant Transformer Adaption) /Trapped /False /arXivID (https://arxiv.org/abs/2506.06105v2) >> endobj 3 0 obj << /Subtype /XML /Type /Metadata /Length 1661 >> stream <?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>

Text-to-LoRA: Instant Transformer Adaption

Rujikorn Charakorn

Edoardo Cetin

Yujin Tang

Robert Tjarko Lange

http://creativecommons.org/licenses/by-nc-sa/4.0/

cs.LG

cs.AI

<?xpacket end="w"?> endstream endobj 4 0 obj << /Dests 7 0 R >> endobj 5 0 obj << /D [ 8 0 R /Fit ] /S /GoTo >> endobj 6 0 obj << /Count 30 /Kids [ 9 0 R 10 0 R 11 0 R 12 0 R 13 0 R ] /Type /Pages >> endobj 7 0 obj << /Kids [ 14 0 R 15 0 R 16 0 R 17 0 R 18 0 R ] /Limits [ (Doc-Start) (table.caption.8) ] >> endobj 8 0 obj << /Annots [ 19 0 R 20 0 R 21 0 R 22 0 R 23 0 R 24 0 R 25 0 R 26 0 R 27 0 R 28 0 R 29 0 R 30 0 R 31 0 R 32 0 R 33 0 R 34 0 R 35 0 R 36 0 R 37 0 R 38 0 R 39 0 R 40 0 R 41 0 R 42 0 R 43 0 R 44 0 R 45 0 R 46 0 R 47 0 R 48 0 R 49 0 R ] /Contents [ 50 0 R 51 0 R ] /MediaBox [ 0 0 612 792 ] /Parent 9 0 R /Resources 52 0 R /Type /Page >> endobj 9 0 obj << /Count 6 /Kids [ 8 0 R 53 0 R 54 0 R 55 0 R 56 0 R 57 0 R ] /Parent 6 0 R /Type /Pages >> endobj 10 0 obj << /Count 6 /Kids [ 58 0 R 59 0 R 60 0 R 61 0 R 62 0 R 63 0 R ] /Parent 6 0 R /Type /Pages >> endobj 11 0 obj << /Count 6 /Kids [ 64 0 R 65 0 R 66 0 R 67 0 R 68 0 R 69 0 R ] /Parent 6 0 R /Type /Pages >> endobj 12 0 obj << /Count 6 /Kids [ 70 0 R 71 0 R 72 0 R 73 0 R 74 0 R 75 0 R ] /Parent 6 0 R /Type /Pages >> endobj 13 0 obj << /Count 6 /Kids [ 76 0 R 77 0 R 78 0 R 79 0 R 80 0 R 81 0 R ] /Parent 6 0 R /Type /Pages >> endobj 14 0 obj << /Kids [ 82 0 R 83 0 R 84 0 R 85 0 R 86 0 R 87 0 R ] /Limits [ (Doc-Start) (cite.dettmers2022gpt3) ] >> endobj 15 0 obj << /Kids [ 88 0 R 89 0 R 90 0 R 91 0 R 92 0 R 93 0 R ] /Limits [ (cite.digre2012shedding) (cite.zhang2018graph) ] >> endobj 16 0 obj << /Kids [ 94 0 R 95 0 R 96 0 R 97 0 R 98 0 R 99 0 R ] /Limits [ (cite.zhang2024mgte) (page.13) ] >> endobj 17 0 obj << /Kids [ 100 0 R 101 0 R 102 0 R 103 0 R 104 0 R 105 0 R ] /Limits [ (page.14) (subfigure.5.1) ] >> endobj 18 0 obj << /Kids [ 106 0 R 107 0 R 108 0 R 109 0 R 110 0 R ] /Limits [ (subfigure.5.2) (table.caption.8) ] >> endobj 19 0 obj << /A << /S /URI /Type /Action /URI (https://github.com/SakanaAI/text-to-lora) >> /Border [ 0 0 0 ] /C [ 0 1 1 ] /H /I /Rect [ 102.342 273.473 270.511 283.824 ] /Subtype /Link /Type /Annot >> endobj 20 0 obj << /A << /S /URI /Type /Action /URI (https://github.com/SakanaAI/text-to-lora) >> /Border [ 0 0 0 ] /C [ 0 1 1 ] /H /I /Rect [ 74.369 261.587 149.089 271.381 ] /Subtype /Link /Type /Annot >> endobj 21 0 obj << /A << /D (cite.wurtz2011neuronal) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 187.92 166.519 236.828 177.462 ] /Subtype /Link /Type /Annot >> endobj 22 0 obj << /A << /D (cite.wurtz2011neuronal) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 239.852 166.519 262.089 177.462 ] /Subtype /Link /Type /Annot >> endobj 23 0 obj << /A << /D (cite.digre2012shedding) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 265.396 166.519 290.437 177.462 ] /Subtype /Link /Type /Annot >> endobj 24 0 obj << /A << /D (cite.digre2012shedding) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 54.444 154.952 100.005 165.507 ] /Subtype /Link /Type /Annot >> endobj 25 0 obj << /A << /D (cite.digre2012shedding) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 102.995 154.952 124.893 165.507 ] /Subtype /Link /Type /Annot >> endobj 26 0 obj << /A << /D (cite.gururangan2020don) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 491.961 591.655 542.437 602.599 ] /Subtype /Link /Type /Annot >> endobj 27 0 obj << /A << /D (cite.gururangan2020don) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 306.444 579.7 327.049 590.644 ] /Subtype /Link /Type /Annot >> endobj 28 0 obj << /A << /D (cite.gururangan2020don) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 329.571 579.7 351.09 590.644 ] /Subtype /Link /Type /Annot >> endobj 29 0 obj << /A << /D (cite.wei2021finetuned) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 353.886 579.7 392.033 590.644 ] /Subtype /Link /Type /Annot >> endobj 30 0 obj << /A << /D (cite.wei2021finetuned) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 394.555 579.7 416.075 590.644 ] /Subtype /Link /Type /Annot >> endobj 31 0 obj << /A << /D (cite.dettmers2022gpt3) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 418.87 579.7 477.341 590.644 ] /Subtype /Link /Type /Annot >> endobj 32 0 obj << /A << /D (cite.dettmers2022gpt3) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 479.863 579.7 501.382 590.644 ] /Subtype /Link /Type /Annot >> endobj 33 0 obj << /A << /D (cite.tay2021scale) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 504.177 579.7 541.241 590.644 ] /Subtype /Link /Type /Annot >> endobj 34 0 obj << /A << /D (cite.tay2021scale) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 306.444 567.745 328.083 578.689 ] /Subtype /Link /Type /Annot >> endobj 35 0 obj << /A << /D (cite.hu2022lora) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 433.103 555.79 469.563 566.734 ] /Subtype /Link /Type /Annot >> endobj 36 0 obj << /A << /D (cite.hu2022lora) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 472.577 555.79 494.814 566.734 ] /Subtype /Link /Type /Annot >> endobj 37 0 obj << /A << /D (cite.bruel2024compress_then_serve) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 349.7 436.627 446.223 447.182 ] /Subtype /Link /Type /Annot >> endobj 38 0 obj << /A << /D (cite.bruel2024compress_then_serve) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 449.195 436.627 471.033 447.182 ] /Subtype /Link /Type /Annot >> endobj 39 0 obj << /A << /D (cite.kim2024cond_lora) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 474.283 436.627 515.695 447.182 ] /Subtype /Link /Type /Annot >> endobj 40 0 obj << /A << /D (cite.kim2024cond_lora) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 518.667 436.627 540.505 447.182 ] /Subtype /Link /Type /Annot >> endobj 41 0 obj << /A << /D (cite.kopiczko2024vera) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 306.444 424.283 366.626 435.227 ] /Subtype /Link /Type /Annot >> endobj 42 0 obj << /A << /D (cite.kopiczko2024vera) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 369.143 424.283 390.663 435.227 ] /Subtype /Link /Type /Annot >> endobj 43 0 obj << /A << /D (cite.ostapenko2024towards_modular_llms) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 475.267 412.328 541.209 423.271 ] /Subtype /Link /Type /Annot >> endobj 44 0 obj << /A << /D (cite.ostapenko2024towards_modular_llms) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 306.444 400.373 328.442 411.316 ] /Subtype /Link /Type /Annot >> endobj 45 0 obj << /A << /D (cite.ha2016hypernetworks) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 415.831 181.383 450.473 192.326 ] /Subtype /Link /Type /Annot >> endobj 46 0 obj << /A << /D (cite.ha2016hypernetworks) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 453.285 181.383 474.805 192.326 ] /Subtype /Link /Type /Annot >> endobj 47 0 obj << /A << /D (cite.wang2022sni) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 389.004 133.562 437.049 144.506 ] /Subtype /Link /Type /Annot >> endobj 48 0 obj << /A << /D (cite.wang2022sni) /S /GoTo >> /Border [ 0 0 0 ] /C [ 0 1 0 ] /H /I /Rect [ 440.282 133.562 462.598 144.506 ] /Subtype /Link /Type /Annot >> endobj 49 0 obj << /A << /S /URI /URI (https://arxiv.org/abs/2506.06105v2) >> /BS << /W 0 >> /NM (fitz-L0) /Rect [ 12 228.23999 32 563.76 ] /Subtype /Link >> endobj 50 0 obj << /Filter /FlateDecode /Length 139 >> stream x?E?? ?0D??????l҂xD?f?M<Ԛ??(???? 23??Ѓ։?W?員?WM ϕ?S?E?????{?o?4rm?vۿ?/|XaMhc?^???y???6??@?4*?|???h?}??'?γ?1?h?hOo8F&? endstream endobj 51 0 obj << /Filter /FlateDecode /Length 5228 >> stream x??[[?۸?~?_??STՈ&.?M*??⍳?$k??:??#aF<?H-Iyv ???7?l?g?N??$@\?F?׍V?-b?k???^?,??????+?d?i[??T[?X????d??/Y$qQ??=??/l??[?????h_??z?Bi???:?x{?Pi?4[?B?I?-?n???.?_?U߬^5???\?4?^?]_?=?.3?e??.u5?޷????<?US/???/W߽}???8?O^CN;Z?5??a V?&??>?_?n??0??fҾٖm9?q8?pq???+?ئ???N?`*4?͚?J?դ?w??l7?????????6?_2??????P,?s??x?E??nn|;?q?D1???????`; ?q:գ??Sq??$z+]?1HL??:Q???M׷庿8???'???l??"?]?;?l??s[?<?o ??m??ޔ$?T?c??ߋ??6??}??%??|??rDž?ivHN?t?8??s?(??&?mZ?і?j??Y?n?????u뉐kx?q??D???lwC??Ǫ?a?ӹ??f?[?j???(?@%?'h?t?(S?Ld?V|??|2??%5#???R|?v??\*T???? ??X??p*???Տ?#?Vzv:u?=??NZ??Ҡ?0Z??#?4N??Fꚝ?=\Z ??n@ ? 2(j5?K~?e$ӄ?q?X??n?V?ø?-??????'??(!K6%~?hi8?`??3}J???*@xN???d?Ҕ?5w?????Ӿ??m????J0ɫ3睸?P??I ??? e"