Common interface for fixed-length and variable-length lists in C#

I am working on library project in which I am reading and writing binary files that have own file format structure. In that structure there are variable-length and fixed-length elements. My approach is to parse byte stream of such files, create corresponding objects, and then get current byte stream and write it back to file. So I need to have byte stream <-> object binding. The problem is very complicated file format (AFP) in which some structures are variable length and some are not. I got stuck on choosing best "list" class for storing byte streams that may be fixed-length or variable-length. My idea is to wrap such byte stream "list" in another class (composition) and allow different external objects to be passed into that class (C# array or List).

namespace TEST
{
    public class ByteStreamList : IList<byte>, IEnumerable<byte>
    {
        protected IList<byte> bytes = null;

        public byte this[int index] {
            get {
                return bytes[index];
            }

            set {
                bytes[index] = value;
            }
        }

        public int Count {
            get {
                return bytes.Count;
            }
        }

        public bool IsReadOnly {
            get {
                return bytes.IsReadOnly;
            }
        }

        public ByteStreamList(IList<byte> bytes)
        {
            this.bytes = bytes;
        }

        public void Add(byte item)
        {
            if (!(bytes is IReadOnlyList<byte>)) {
                bytes.Add(item);
            }
        }

        public void Clear()
        {
            if (!(bytes is IReadOnlyList<byte>)) {
                bytes.Clear();
            }
        }

        public bool Contains(byte item)
        {
            if (!(bytes is IReadOnlyList<byte>)) {
                return bytes.Contains(item);
            }

            return false;
        }

    // ... rest of IList<byte> and IEnumerable<byte> interface ///
}

namespace Test_CA
{
    public class Program
    {
        public static void Main(string[] args)
        {
            ByteStreamList fixedLengthByteList = new ByteStreamList(new byte[] { 0xAA, 0xBB, 0xCC });
            ByteStreamList variableLengthByteList = new ByteStreamList(new List<byte>() { 0xDD, 0xEE, 0xFF });
        }
    }
}

With that approach (using IList<> class) I would need to provide some access to byte stream via IList<> methods. So I could let my wrapper class implement IList<>. That approach forces me to check if underlying "list" class is IReadOnlyList or not. If not, methods like Add() Clear() would be allowed. I know it is very bad approach to use List<> because if something will require that type I will not pass any other list/container except List<> .

Is it good approach with checking everytime if "list" is readonly?