使用 Protobuf-net 反序列化二进制文件时的线型无效

Posted 2023-02-16

【中文标题】使用 Protobuf-net 反序列化二进制文件时的线型无效

【英文标题】：Invalid wire-type when deserializing a binary file with Protobuf-net

【发布时间】：2014-04-07 11:50:13

【问题描述】：

什么可能导致这个错误？

Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see http://***.com/q/2152978/23354

我按照上面的帮助链接，但设置 FileMode.Truncate 没有帮助...

问题总是发生在一个被反序列化的特定文件的相同位置。我已经多次重新创建该文件。

    [ProtoContract]
    public class TickRecord
    
        [ProtoMember(1, DataFormat = DataFormat.FixedSize)]
        public DateTime DT;
        [ProtoMember(2)]
        public double BidPrice;
        [ProtoMember(3)]
        public double AskPrice;
        [ProtoMember(4, DataFormat = DataFormat.FixedSize)]
        public int BidSize;
        [ProtoMember(5, DataFormat = DataFormat.FixedSize)]
        public int AskSize;

        public TickRecord(DateTime DT, double BidPrice, double AskPrice, int BidSize, int AskSize)
        
            this.DT = DT;
            this.BidPrice = BidPrice;
            this.AskPrice = AskPrice;
            this.BidSize = BidSize;
            this.AskSize = AskSize;

        

        public TickRecord()
        
        

为乔恩·斯基特编辑

这行得通。但是当我使用我的真实数据（我无法分享）时，我的问题就出现了。我反序列化的大多数文件都可以正常工作，但有一个文件不能......

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using ProtoBuf;
using System.IO;
using System.Diagnostics;

namespace BinTest3


    
    public partial class Form1 : Form
    
        public Form1()
        
            InitializeComponent();
        

        private void Serialize_Click(object sender, EventArgs e)
        

            FileStream outBin = null;

            string binFileName = @"C:\binfile.dft";
            outBin = File.Create(binFileName, 2048, FileOptions.None);

            DateTime d = DateTime.Now;

            TickRecord tr = new TickRecord(d, 1.02, 1.03,200,300);

            for (int i =0; i < 20000000; i++)
            
                tr.BidPrice += 1;
                Serializer.SerializeWithLengthPrefix(outBin, tr, PrefixStyle.Base128);
            

            outBin.Close();
            label1.Text = "Done ";
        

        private void Deserialize_Click(object sender, EventArgs e)
        
            Stopwatch sw = new Stopwatch();
            sw.Start();

            FileStream fs;
            string binFileName = @"C:\binfile.dft";

            byte[] data = File.ReadAllBytes(binFileName);
            MemoryStream ms = new MemoryStream(data);

            long skipRate =10;
            int count = 0;
            TickRecord tr;

            long skip = (38*skipRate);
            try
            
                
                while ((tr = Serializer.DeserializeWithLengthPrefix<TickRecord>(ms, PrefixStyle.Base128)) != null) //fs.Length > fs.Position)
                
                    count++;

                   
                    ms.Seek(skip, SeekOrigin.Current);

                
            
            catch (Exception)
            

            

            ms.Close();

            sw.Stop();
            label1.Text = "Time taken: " + sw.Elapsed + " Count: " + count.ToString("n0");

        
    


    [ProtoContract]
    public class TickRecord
    

        [ProtoMember(1, DataFormat = DataFormat.FixedSize)]
        public DateTime DT;
        [ProtoMember(2)]
        public double BidPrice;
        [ProtoMember(3)]
        public double AskPrice;
        [ProtoMember(4, DataFormat = DataFormat.FixedSize)]
        public int BidSize;
        [ProtoMember(5, DataFormat = DataFormat.FixedSize)]
        public int AskSize;

        public TickRecord()
        

        

        public TickRecord(DateTime DT, double BidPrice, double AskPrice, int BidSize, int AskSize)
        
            this.DT = DT;
            this.BidPrice = BidPrice;
            this.AskPrice = AskPrice;
            this.BidSize = BidSize;
            this.AskSize = AskSize;

        



    

编辑 2 问题在序列化中 - 它不是我想象的固定大小：

private void Once_Click(object sender, EventArgs e)

    FileStream outBin = null;

    string binFileName = @"C:\binfile.dft";
    outBin = File.Create(binFileName, 2048, FileOptions.None);

    DateTime d = DateTime.Now;
    long lastPosition = 0;

    TickRecord tr = new TickRecord(d, 1.02, 1.03, 200, 300);
    Serializer.SerializeWithLengthPrefix(outBin, tr, PrefixStyle.Base128);
    Console.WriteLine("outBin.Position: " + outBin.Position + " lastPosition: " + lastPosition + " diff: " + (outBin.Position-lastPosition));
    lastPosition = outBin.Position;
    tr = new TickRecord(d, 1.02, 0.0, 200, 300);
    Serializer.SerializeWithLengthPrefix(outBin, tr, PrefixStyle.Base128);
    Console.WriteLine("outBin.Position: " + outBin.Position + " lastPosition: " + lastPosition + " diff: " + (outBin.Position - lastPosition));


    outBin.Close();
    label1.Text = "Done ";





  [ProtoContract]
  public class TickRecord
  

    [ProtoMember(1, DataFormat = DataFormat.FixedSize)]
    public DateTime DT;
    [ProtoMember(2, DataFormat = DataFormat.FixedSize)]
    public double BidPrice;
    [ProtoMember(3, DataFormat = DataFormat.FixedSize)]
    public double AskPrice;
    [ProtoMember(4, DataFormat = DataFormat.FixedSize)]
    public int BidSize;
    [ProtoMember(5, DataFormat = DataFormat.FixedSize)]
    public int AskSize;

public TickRecord()




public TickRecord(DateTime DT, double BidPrice, double AskPrice, int BidSize, int AskSize)

    this.DT = DT;
    this.BidPrice = BidPrice;
    this.AskPrice = AskPrice;
    this.BidSize = BidSize;
    this.AskSize = AskSize;

这给出了：

outBin.Position: 38 lastPosition: 0 diff: 38
outBin.Position: 67 lastPosition: 38 diff: 29

所以看起来变量的大小在为零时是不同的。

如何使零双精度与非零双精度具有相同的大小？

【问题讨论】：

您能否展示一个简短但完整的程序来说明问题？鉴于您之前在流中跳过，我想知道您是否跳到了错误的地方......

@JonSkeet 这是一个 SSCCE。它没有显示实际错误，因为我无法上传大数据文件。如果您愿意 - 我可以直接通过电子邮件发送给您...

您确定指定DataFormat = DataFormat.FixedSize 是严格要求的吗？我相当有信心反序列化器可以自己找到整数的大小。在我看来，这只会造成麻烦。

@aevitas：这是为了确保它是一个固定大小的整数，而不是一个可变长度的整数。这里的背景是 OP 真的想要一个固定的记录长度。

您有多确定 38 是正确的数字？你能用少于 2000 万条记录重现这个问题吗？它是否总是以相同的数字失败，如果您以该出价开始它，它会立即失败吗？稍后会尝试您的代码（现在不能这样做），但是您可以将其固定的越多越好。

【参考方案1】：

你需要 IsRequired 属性。

零值 double 和 int 字段的序列化方式与非零字段不同。所以虽然我指定了 FixedSize - 但不是。

以下创建一个真正的 FixedSize 记录：

 [ProtoContract]
    public class TickRecord
    
        [ProtoMember(1, DataFormat = DataFormat.FixedSize, IsRequired = true)]
        public DateTime DT;
        [ProtoMember(2, DataFormat = DataFormat.FixedSize, IsRequired = true)]
        public double BidPrice;
        [ProtoMember(3, DataFormat = DataFormat.FixedSize, IsRequired = true)]
        public double AskPrice;
        [ProtoMember(4, DataFormat = DataFormat.FixedSize, IsRequired = true)]
        public int BidSize;
        [ProtoMember(5, DataFormat = DataFormat.FixedSize, IsRequired = true)]
        public int AskSize;