加载预训练模型:OSError: Can‘t load config for XXX Make sure that:XXXis a correct model identifier listed on |
您所在的位置：网站首页 › 报错7b › 加载预训练模型:OSError: Can‘t load config for XXX Make sure that:XXXis a correct model identifier listed on |
|
1.背景
从huggingface模型库中，加载一些预训练模型，做fine-tuning。 2.错误详情demo.py代码结构： from transformers import AutoTokenizer, AutoModelForMaskedLM,BertTokenizer, BertModel, BertForMaskedLM, pipeline import torch tokenizer = BertTokenizer.from_pretrained("uer/chinese_roberta_L-12_H-128") model = BertForMaskedLM.from_pretrained("uer/chinese_roberta_L-12_H-128") text = "我是一个学生。" tokenized_text = tokenizer.tokenize(text) masked_index = 2 # tokenized_text[masked_index] = '[MASK]' indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text) tokens_tensor = torch.tensor([indexed_tokens]) encoded_input = tokenizer(text, return_tensors='pt') output = model(tokens_tensor) print(output)预训练模型加载报错： OSError: Can't load config for 'uer/chinese_roberta_L-12_H-512'. Make sure that: - 'uer/chinese_roberta_L-12_H-512' is a correct model identifier listed on 'https://huggingface.co/models' - or 'uer/chinese_roberta_L-12_H-512' is the correct path to a directory containing a config.json file错误的意思是没有这个模型。 3.排查以及解决 排查1：是否是transformers的版本低?显然不是，我的版本是3.4.0，稳定版本。4.X版本目前是release，不太稳定，最开始遇到该问题时，我升级了transformers的版本，然后各种报错，之前能加载的base预训练模型，升级后，也加载不出来了。所以此时直接排除transformers版本问题。 排查2：可能是下载的文件有损失。遇到过这个错误，偶然间，从开发机上，拉模型后，本地预测时，报错这个错，当时问了运维，同学反馈不能拉文件，也就是说拉下来的文件是有损的，因此我这边想着可能是拉文件有损。那么接下来的问题是：如果让模型文件无损的拉下来。 那么严格按照huggingface网站上给出的下载流程进行下载，会如何？ # How to use from the 🤗/transformers library: # Copy to clipboard, # this method can not do well from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("uer/chinese_roberta_L-12_H-128") model = AutoModelForMaskedLM.from_pretrained("uer/chinese_roberta_L-12_H-128") # Or just clone the model repo git lfs install git clone https://huggingface.co/bert-base-chinese # if you want to clone without large files – just their pointers # prepend your git clone with the following env var: GIT_LFS_SKIP_SMUDGE=1最开始直接git clone XXX,没有按要求安装lfs。 严格执行一遍： git lfs install 报错： git: 'lfs' is not a git command. See 'git --help'.解决(参考1)： MacOS (Using Homebrew) $ brew update $ brew install git-lfs然后再来一遍： git lfs install git clone https://huggingface.co/bert-base-chinese # if you want to clone without large files – just their pointers # prepend your git clone with the following env var: GIT_LFS_SKIP_SMUDGE=1下载预训练模型文件，放到你的项目工程下，再运行demo，输出成功： /Users/wang/anaconda3/envs/torch/bin/python /Users/wang/bigdata/feedrec/LM_embedding/demo.py (tensor([[[-18.8498, -11.7181, -10.0634, ..., -10.0189, -6.4434, -8.5150], [-20.6099, -13.5073, -11.4595, ..., -12.6746, -10.1947, -11.0346], [-18.4483, -12.0095, -10.5682, ..., -12.0273, -12.5031, -9.2698], ..., [-15.2799, -10.9625, -11.5090, ..., -12.6011, -11.9219, -10.2075], [-15.7109, -10.3530, -11.0650, ..., -11.4598, -8.6766, -9.5837], [-17.7121, -10.3701, -12.0171, ..., -14.6374, -12.2308, -11.2659]]], grad_fn=),) Some weights of BertModel were not initialized from the model checkpoint at ./chinese_roberta_L-12_H-128 and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of BertModel were not initialized from the model checkpoint at ./chinese_roberta_L-12_H-128 and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. [[[0.0948249027132988, -0.6719890832901001, -0.6491043567657471, .....]]] --a[0]: 9 ---a[0][0]: 128 Process finished with exit code 0tips： 1.该模型是最近202012月上传的，也就是说，上传者的环境可能用的比较新，导致如果git版本过低，则不能无损下载该文件。 2.不要盲目升级最新版本的transformers，release版本，不稳定，可能会导致v3和v4版本不兼容，以前的项目也会运行失败。
参考： 1.https://stackoverflow.com/questions/48734119/git-lfs-is-not-a-git-command-unclear/48734334
|
| CopyRight 2018-2019 办公设备维修网 版权所有 豫ICP备15022753号-3 |