Got NativeDbErr(IncorrectInputData) when remove or update

[tklam]

Hi, thanks for implementing this powerful db in Rust. I have got a model:

#[derive(Serialize, Deserialize, Debug)]
#[native_model(id = 1, version = 1)]
#[native_db]
pub struct Model {
  #[primary_key]
  pub name: String,
  pub networks: Vec<Network>,
}

The struct Network is defined in this way:

#[derive(Serialize, Deserialize, Debug)]
pub struct Network {
    name: String,
    config: Rc<Config>, // Config is a struct containing many String members
    inputs: Vec<u32>,
    optional_input: Option<u32>,
    input_to_value: HashMap<u32, u32>,
    registry: Rc<RefCell<Registry>>,
    cache: Arc<RwLock<Cache>>,  // Cache is a struct with many HashSet and HashMap members
}

#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Beh {
    members: Vec<u32>,
    beh_value: u64,
}

#[derive(Serialize, Deserialize, Debug, Default)]
pub struct Registry {
    pair_to_beh: HashMap<(u32, u32), Beh>,
}

This model could be saved into a file without any problem. But when I read back the model and tried to remove(), update(), or upsert(), native_db threw an error IncorrectInputData. It seems the error was caused by the check current_item == item.value in concrete_remove() in transaction/internal/rw_transaction.rs (line 171):

161     pub(crate) fn concrete_remove(
162         &self,
163         model: Model,
164         item: Input,
165     ) -> Result<(WatcherRequest, Output)> {
166         let keys = &item.secondary_keys;
167         {
168             let mut table: redb::Table<Key, &[u8]> = self.get_primary_table(&model)?;
169             let result = if let Some(current_item) = table.remove(&item.primary_key)? {
170                 let current_item = current_item.value();
171                 if current_item == item.value {
172                     Ok(())
173                 } else {
174                     Err(Error::IncorrectInputData {
175                         value: current_item.to_vec(),
176                     })
177                 }
178             } else {
179                 Err(Error::KeyNotFound {
180                     key: item.primary_key.as_slice().to_vec(),
181                 })
182             };
183             if let Err(Error::IncorrectInputData { ref value }) = result {
184                 table.insert(&item.primary_key, value.as_slice())?;
185             }
186             result?;
187         }

So my questions are:

• Did I use the library correctly? I am very new to Rust and native_db
• Is it the HashSet and HashMap that made current_item == item.value false?
• Can this error be ignored by skipping the check?
• Do you recommend using native_db to store this kind of data structures?

Thank you so much

[vincent-herlemont]

@tklam Thank you for your message. I haven't had the time to look into it yet (I'll try during the week), but at first glance, the serialization of HashSet and HashMap might pose a problem because the order is not guaranteed.

You can run a test with native_model by simply trying to encode and decode your structure. And maybe try using a different serializer than the default one. And maybe use ordermap.

[tklam]

Thanks a lot. I tried encoding and decoding the above structs. native_db worked out of the box using the default serializer. There was no issue at all with HashSet and HashMap. Perhaps my setup was wrong; or it might because the structs listed here are in fact the simplified version of the actual structs in my project that they cannot reflect the real situation. Let me check it thoroughly.

[jokeyrhyme]

I've been experimenting with a similar issue, and for me, it looks like removing the HashMap fields from my model/record type fixes it

Beyond the primitive types, my model/record type has Vec and Option fields, and those seem to work just fine

But as soon as I add the HashMap field, I can't do upsert(), etc

[agersant]

I had the same issue and I can confirm replacing usage of HashMap by BTreeMap is a successful workaround.

Also of note, I was upgrading from an earlier version of native_db (6daae43) where the same model with HashMap was working fine.

[rustmailer]

I believe this check may be somewhat unreasonable. For instance, if the frontend request content is constructed into a new object, even with identical data, the serialized results might differ due to varying internal collection element orders. In such cases, updates should still be permitted. It would be better if the serialization tool could ensure consistent results or provide an option to skip this check. This seems like a potential issue that could be addressed.

[lbarnkow]

I just stumbled over this very issue, as well. I had a HashMap as part of my model struct. Replacing it with BTreeMap, as suggested above, solved the issue. I'd suggest to at least put a disclaimer into the docs and possibly the Tutorial to warn about types that may have non-deterministic ordering of elements/attributes when serializing/deserializing. :)