Added weightless cache attributes for intel GPU plugin. #31468
Conversation
susbhere
force-pushed
the
gpu_weighless_caching
branch
3 times, most recently
from
f108dbe to
c9f955c
Compare
susbhere
force-pushed
the
gpu_weighless_caching
branch
from
c9f955c to
84e65dd
Compare
|
build_jenkins |
| } | ||
| } else if (auto ti = ov::as_type<const ov::op::v0::TensorIterator>(node.get())) { | ||
| auto ti_body = ti->get_body(); | ||
| fill_offset_to_constant_map(ti_body); |
There was a problem hiding this comment.
| fill_offset_to_constant_map(ti_body); | |
| fill_offset_to_constant_map(ti_body, cache_attr); |
There is no logic for handling TensorIterator in set_weightless_cache_attributes(), so either its body should be ignored here, or the logic should be added to the corresponding functions
There was a problem hiding this comment.
There are unit test failure with suggested changes. 4 tests out of 193 weightless cache related tests are failing. One such test is below one.
[ FAILED ] smoke_CheckWeightlessCacheAccuracy/CheckWeightlessCacheAccuracy.TiWithLstmCell/import_api=compile_model_do_encryption=1_inference_mode=f32_model_dtype=f32, where GetParam() = (4-byte object <02-00 00-00>, true, f32, f32) (273 ms)
There was a problem hiding this comment.
@susbhere, did you check the tests with proper TensorIterator handling in set_weightless_cache_attributes() / create_weightless_cache_attributes() functioncs?
There was a problem hiding this comment.
@sshlyapn, I have run below command and saw the failures. 4 out of 193 tests failed. Those are not failing with current state of the PR.
ov_gpu_func_tests.exe --gtest_filter=Weightless
There was a problem hiding this comment.
@susbhere, my point is that either this TensorIterator-related condition is unnecessary, or TensorIterator should be properly handled during runtime attribute configuration. Currently, only constants are processed there:
for (const auto& node : model->get_ordered_ops()) {
if (ov::op::util::is_constant(node)) {
// Offset behaves as a unique key for each constant. Size = 1 is used as dummy.
cache_attr_map->emplace(node->get_instance_id(), ov::WeightlessCacheAttribute(1, offset++, node->get_element_type()));
}
}
src/plugins/intel_gpu/include/intel_gpu/runtime/internal_properties.hpp
Outdated
Show resolved
Hide resolved
| @@ -1860,14 +1860,20 @@ void program::save(cldnn::BinaryOutputBuffer& ob) const { | |||
| } | |||
| } | |||
|
|
|||
| void program::load(cldnn::BinaryInputBuffer& ib, std::shared_ptr<const ov::Model> model_ptr) { | |||
| void program::load(cldnn::BinaryInputBuffer& ib, | |||
There was a problem hiding this comment.
[random spot]
Could you please add some functional tests for this caching mechanism?
susbhere
force-pushed
the
gpu_weighless_caching
branch
from
84e65dd to
e4a48d2
Compare
Renaming variables and minor updates as per review comments.
Weightless cache attributes are added when the weights are not coming from bin file. That happens for non-IR inputs like ORT.
https://jira.devtools.intel.com/browse/CVS-167691