Assertion `!node_algorithms::inited(this->priv_value_traits().to_node_ptr(value))' failed

This issue has been created since 2021-12-31.

Installation details
Kernel version: 5.11.0-1022-aws
Scylla version (or git commit hash): 4.7.dev-0.20211223.f7b8b809d with build-id 77a4b7dabbcda8796546b2a14f1167eacfbdc222
Cluster size: 3 nodes (i3.4xlarge)
Scylla running with shards number (live nodes):
gemini-1tb-10h-master-db-node-b282f96d-1 (3.250.40.207 | 10.0.1.252): 14 shards
gemini-1tb-10h-master-db-node-b282f96d-2 (63.35.251.194 | 10.0.2.220): 14 shards
gemini-1tb-10h-master-db-node-b282f96d-3 (54.154.146.231 | 10.0.1.225): 14 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-00687fba1ec0e4d1e (aws: eu-west-1)

Gemini command:
/$HOME/gemini -d --duration 8h --warmup 2h -c 50 -m mixed -f --non-interactive --cql-features normal --max-mutation-retries 5 --max-mutation-retries-backoff 500ms --async-objects-stabilization-attempts 5 --async-objects-stabilization-backoff 500ms --replication-strategy "{'class': 'SimpleStrategy', 'replication_factor': '3'}" --oracle-replication-strategy "{'class': 'SimpleStrategy', 'replication_factor': '1'}" --test-cluster=10.0.1.252,10.0.2.220,10.0.1.225 --outfile /home/centos/gemini_result_4283cae0-357e-4064-8193-227fadd63582.log --seed 93 --oracle-cluster=10.0.0.212
Gemini version: 1.7.5

Test: gemini-1tb-10h
Test name: gemini_test.GeminiTest.test_load_random_with_nemesis
Test config file(s):

Issue description

====================================

During Gemini run got coredump on node 3:

                                                                         Total: 106 permits with 49 count and 20M memory resources
Dec 24 10:58:02 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:  [shard 6] reader_concurrency_semaphore - (rate limiting dropped 221 similar messages) Semaphore _read_concurrency_sem with 23/100 count and 9733668/171001774 memory resources: timed out,>
                                                                         permits        count        memory        table/description/state
                                                                         22        22        9M        ks1.table1_mv_0/data-query/active/blocked
                                                                         1        1        16K        ks1.table1_mv_0/data-query/active/used
                                                                         7        0        0B        ks1.table1/data-query/waiting
                                                                         57        0        0B        ks1.table1_mv_0/data-query/waiting
                                                                         
                                                                         87        23        9M        total
                                                                         
                                                                         Total: 87 permits with 23 count and 9M memory resources
2021-12-24T10:58:06+00:00 gemini-1tb-10h-master-db-node-b282f96d-3 !    INFO | scylla: /usr/include/boost/intrusive/list.hpp:1307: boost::intrusive::list_impl::iterator boost::intrusive::list_impl<boost::intrusive::bhtraits<logalloc::segment_descriptor, boost::intrusive::list_node_traits<void *>, boost::intrusive::safe_link, boost::intrusive::dft_tag, 1>, unsigned long, false, void>::iterator_to(boost::intrusive::list_impl::reference) [ValueTraits = boost::intrusive::bhtraits<logalloc::segment_descriptor, boost::intrusive::list_node_traits<void *>, boost::intrusive::safe_link, boost::intrusive::dft_tag, 1>, SizeType = unsigned long, ConstantTimeSize = false, HeaderHolder = void]: Assertion `!node_algorithms::inited(this->priv_value_traits().to_node_ptr(value))' failed.
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]: Aborting on shard 8.
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]: Backtrace:
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x4066d58
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x4097db2
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x7fe141177a1f
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   /opt/scylladb/libreloc/libc.so.6+0x3d2a1
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   /opt/scylladb/libreloc/libc.so.6+0x268a3
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   /opt/scylladb/libreloc/libc.so.6+0x26788
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   /opt/scylladb/libreloc/libc.so.6+0x35a15
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x144af05
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x16dfd54
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x16df6c4
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x13bdff2
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x14509fd
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x145cb18
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x1458663
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x1447fdb
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x14667c7
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x17c0ff0
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x17bdabe
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x17c60c0
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x4077f64
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x4079357
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x40982b5
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   0x404a57a
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   /opt/scylladb/libreloc/libpthread.so.0+0x9298
Dec 24 10:58:06 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[209091]:   /opt/scylladb/libreloc/libc.so.6+0x100352
Dec 24 11:02:11 gemini-1tb-10h-master-db-node-b282f96d-3 scylla[214450]: Scylla version 4.7.dev-0.20211223.f7b8b809d with build-id 77a4b7dabbcda8796546b2a14f1167eacfbdc222 starting ...

translated backtrace

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:760
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:790
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:802
 (inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3675
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3657
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3653
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
logalloc::lsa_buffer::~lsa_buffer() at logalloc.cc:?
 (inlined by) ~lsa_buffer at ./utils/logalloc.cc:1876
~cached_page at ././utils/cached_file.hh:119
 (inlined by) ~data at ././utils/bptree.hh:1946
 (inlined by) void allocation_strategy::destroy<bplus::data<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0> >(bplus::data<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>*) at ././utils/allocation_strategy.hh:182
 (inlined by) void bplus::data<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::destroy<void (&)(cached_file::cached_page*) noexcept>(bplus::data<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>&, void (&)(cached_file::cached_page*) noexcept) at ././utils/bptree.hh:1932
 (inlined by) bplus::tree<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::iterator bplus::tree<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::iterator::erase_and_dispose<void (&)(cached_file::cached_page*) noexcept>(void (&)(cached_file::cached_page*) noexcept, cached_file::page_idx_less_comparator) at ././utils/bptree.hh:808
 (inlined by) bplus::tree<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::iterator::erase(cached_file::page_idx_less_comparator) at ././utils/bptree.hh:813
cached_file::cached_page::on_evicted() at ././utils/cached_file.hh:475
lru::evict() at ././utils/lru.hh:91
 (inlined by) operator() at ./row_cache.cc:83
 (inlined by) decltype(auto) with_allocator<cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0::operator()() const::{lambda()#1}>(allocation_strategy&, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0::operator()() const::{lambda()#1}&&) at ././utils/allocation_strategy.hh:328
 (inlined by) operator() at ./row_cache.cc:70
 (inlined by) seastar::memory::reclaiming_result std::__invoke_impl<seastar::memory::reclaiming_result, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&>(std::__invoke_other, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<seastar::memory::reclaiming_result, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&>, seastar::memory::reclaiming_result>::type std::__invoke_r<seastar::memory::reclaiming_result, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&>(cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:114
 (inlined by) std::_Function_handler<seastar::memory::reclaiming_result (), cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:291
std::function<seastar::memory::reclaiming_result ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:560
 (inlined by) logalloc::region_impl::evict_some() at ./utils/logalloc.cc:1849
logalloc::reclaim_from_evictable(logalloc::region_impl&, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2089
logalloc::tracker::impl::compact_and_evict_locked(unsigned long, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2356
logalloc::tracker::impl::compact_and_evict(unsigned long, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2291
 (inlined by) logalloc::segment_pool::allocate_segment(unsigned long) at ./utils/logalloc.cc:975
logalloc::segment_pool::refill_emergency_reserve() at ./utils/logalloc.cc:988
 (inlined by) logalloc::allocating_section::reserve() at ./utils/logalloc.cc:2652
decltype(auto) logalloc::allocating_section::with_reserve<logalloc::allocating_section::operator()<sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()#1}>(logalloc::region&, sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()#1}&&)::{lambda()#1}>(logalloc::allocating_section::operator()<sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()#1}>(logalloc::region&, sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()#1}&&)::{lambda()#1}) at ././utils/logalloc.hh:780
 (inlined by) decltype(auto) logalloc::allocating_section::operator()<sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()#1}>(logalloc::region&, sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&)::{lambda()#1}&&) at ././utils/logalloc.hh:831
 (inlined by) sstables::index_consumer::consume_entry(sstables::parsed_partition_index_entry&&) at ./sstables/index_reader.hh:92
 (inlined by) sstables::index_consume_entry_context<sstables::index_consumer>::process_state(seastar::temporary_buffer<char>&) at ./sstables/index_reader.hh:282
data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::process(seastar::temporary_buffer<char>&) at ././sstables/consumer.hh:560
 (inlined by) data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::operator()(seastar::temporary_buffer<char>) at ././sstables/consumer.hh:594
 (inlined by) seastar::future<seastar::consumption_result<char> > std::__invoke_impl<seastar::future<seastar::consumption_result<char> >, sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> >(std::__invoke_other, sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char>&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::__invoke_result<sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> >::type std::__invoke<sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> >(sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char>&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:96
 (inlined by) std::result_of<sstables::index_consume_entry_context<sstables::index_consumer>& (seastar::temporary_buffer<char>&&)>::type std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >::operator()<seastar::temporary_buffer<char> >(seastar::temporary_buffer<char>&&) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/refwrap.h:349
 (inlined by) operator() at ././seastar/include/seastar/core/iostream-impl.hh:222
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}&>(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&) at ././seastar/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}&>(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&) at ././seastar/include/seastar/core/future.hh:2166
 (inlined by) seastar::internal::repeater<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>::run_and_dispose() at ././seastar/include/seastar/core/loop.hh:71
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2354
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2763
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2932
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4128
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_99&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_99&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_99&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_99&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_99&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_99>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:291
std::function<void ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:560
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Coredump

CoreDumpEvent Severity.ERROR) period_type=one-time event_id=480cba47-446e-439e-9a1b-e30baef44795 node=Node gemini-1tb-10h-master-db-node-b282f96d-3 [54.154.146.231 | 10.0.1.225] (seed: False)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.ca68faa9ae854b6bb1c94e3741583354.209091.1640343486000000000000/core.scylla.113.ca68faa9ae854b6bb1c94e3741583354.209091.1640343486000000000000.gz
backtrace=           PID: 209091 (scylla)
UID: 113 (scylla)
GID: 119 (scylla)
Signal: 6 (ABRT)
Timestamp: Fri 2021-12-24 10:58:06 UTC (29min ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
Unit: scylla-server.service
Slice: scylla-server.slice
Boot ID: ca68faa9ae854b6bb1c94e3741583354
Machine ID: 740adf4730354ea7bc7fdeb7bd0b3164
Hostname: gemini-1tb-10h-master-db-node-b282f96d-3
Storage: /var/lib/systemd/coredump/core.scylla.113.ca68faa9ae854b6bb1c94e3741583354.209091.1640343486000000000000
Message: Process 209091 (scylla) of user 113 dumped core.
Stack trace of thread 209099:
#0  0x00007fe1405bc2a2 raise (libc.so.6 + 0x3d2a2)
#1  0x00007fe1405a5950 abort (libc.so.6 + 0x26950)
#2  0x00007fe1405a5789 __assert_fail_base.cold (libc.so.6 + 0x26789)
#3  0x00007fe1405b4a16 __assert_fail (libc.so.6 + 0x35a16)
#4  0x000000000144af06 _ZN8logalloc10lsa_bufferD1Ev (scylla + 0x124af06)
#5  0x00000000016dfd55 _ZN5bplus4treeImN11cached_file11cached_pageENS1_24page_idx_less_comparatorELm12ELNS_10key_searchE0ELNS_10with_debugE0EE8iterator5eraseES3_ (scylla + 0x14dfd55)
#6  0x00000000016df6c5 _ZN11cached_file11cached_page10on_evictedEv (scylla + 0x14df6c5)
#7  0x00000000013bdff3 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x11bdff3)
#8  0x00000000014509fe _ZN8logalloc11region_impl10evict_someEv (scylla + 0x12509fe)
#9  0x000000000145cb19 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x125cb19)
#10 0x0000000001458664 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x1258664)
#11 0x0000000001447fdc _ZN8logalloc12segment_pool16allocate_segmentEm (scylla + 0x1247fdc)
#12 0x00000000014667c8 _ZN8logalloc18allocating_section7reserveEv (scylla + 0x12667c8)
#13 0x00000000017c0ff1 _ZN8sstables27index_consume_entry_contextINS_14index_consumerEE13process_stateERN7seastar16temporary_bufferIcEE (scylla + 0x15c0ff1)
#14 0x00000000017bdabf _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables27index_consume_entry_contextINS4_14index_consumerEEEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x15bdabf)
#15 0x00000000017c60c1 _ZN7seastar8internal8repeaterIZNS_12input_streamIcE7consumeISt17reference_wrapperIN8sstables27index_consume_entry_contextINS6_14index_consumerEEEEEENS_6futureIvEEOT_EUlvE_E15run_and_disposeEv (scylla + 0x15c60c1)
#16 0x0000000004077f65 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3e77f65)
#17 0x0000000004079358 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79358)
#18 0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#19 0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#20 0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#21 0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209111:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209108:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209112:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209116:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209105:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209095:
#0  0x0000000004087a04 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3e87a04)
#1  0x00000000040b533b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3eb533b)
#2  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#3  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#4  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#5  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#6  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#7  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209110:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209100:
#0  0x0000000002b438e2 _ZN5utils11murmur_hash13hash3_x64_128IN20legacy_compound_viewI13compound_typeIL14allow_prefixes0EEE8iteratorEEEvT_jmRSt5arrayImLm2EE (scylla + 0x29438e2)
#1  0x0000000002b4345a _ZNK3dht19murmur3_partitioner9get_tokenERK6schema18partition_key_view (scylla + 0x294345a)
#2  0x000000000193ea44 _ZN8sstables2mx17mp_row_consumer_m23consume_partition_startENS_8key_viewENS_13deletion_timeE (scylla + 0x173ea44)
#3  0x0000000001965bf2 _ZN8sstables2mx27data_consume_rows_context_m16do_process_stateEv.resume (scylla + 0x1765bf2)
#4  0x000000000195308e _ZN13data_consumer24continuous_data_consumerIN8sstables2mx27data_consume_rows_context_mEEclEN7seastar16temporary_bufferIcEE (scylla + 0x175308e)
#5  0x000000000195285c _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables2mx27data_consume_rows_context_mEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x175285c)
#6  0x00000000019575ba _ZZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEvENKUlvE0_clEv (scylla + 0x17575ba)
#7  0x000000000194f8b8 _ZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEv (scylla + 0x174f8b8)
#8  0x00000000019301bc _ZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x17301bc)
#9  0x00000000019327dd _ZThn64_N8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x17327dd)
#10 0x00000000015c1db0 _ZN22mutation_reader_mergerclEv (scylla + 0x13c1db0)
#11 0x00000000015fb13d _ZN24mutation_fragment_mergerI22mutation_reader_mergerEclEv (scylla + 0x13fb13d)
#12 0x00000000015fa9d1 _ZN7seastar6repeatIZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvEUlvE_EENS_6futureIvEEOT_ (scylla + 0x13fa9d1)
#13 0x00000000015f9c57 _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x13f9c57)
#14 0x000000000169495f _ZN7seastar8futurizeINS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEE6invokeIRZN23flat_mutation_reader_v24impl16consume_pausableIZ15downgrade_to_v1S8_EN19transforming_reader8consumerEEENS1_IvEET_EUlvE_JEEES5_OSE_DpOT0_ (scylla + 0x149495f)
#15 0x00000000016916f1 _ZZ15downgrade_to_v123flat_mutation_reader_v2EN19transforming_reader11fill_bufferEv (scylla + 0x14916f1)
#16 0x0000000001a25a66 _ZZZN8sstables10compaction7consumeEvENUl20flat_mutation_readerE_clES1_ENUlvE_clEv (scylla + 0x1825a66)
#17 0x0000000001a25354 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction7consumeEvENUl20flat_mutation_readerE_clES7_EUlvE_JEEENS_8futurizeINSt13invoke_resultIT_JDpT0_EE4typeEE4typeENS_17thread_attributesEOSC_DpOSD_EUlvE_E4callEPKS2_ (scylla + 0x1825354)
#18 0x00000000042e78a2 _ZN7seastar14thread_context4mainEv (scylla + 0x40e78a2)
Stack trace of thread 209098:
#0  0x000000000407b7a6 _ZN7seastar17smp_message_queue20flush_response_batchEv (scylla + 0x3e7b7a6)
#1  0x0000000004087a4d _ZN7seastar3smp11poll_queuesEv (scylla + 0x3e87a4d)
#2  0x00000000040b533b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3eb533b)
#3  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#4  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#5  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#6  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#7  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#8  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209091:
#0  0x0000000004087a22 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3e87a22)
#1  0x00000000040b533b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3eb533b)
#2  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#3  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#4  0x000000000407859d _ZN7seastar7reactor3runEv (scylla + 0x3e7859d)
#5  0x0000000004017398 _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x3e17398)
#6  0x0000000004016837 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x3e16837)
#7  0x0000000000f9e12f main (scylla + 0xd9e12f)
#8  0x00007fe1405a6b75 __libc_start_main (libc.so.6 + 0x27b75)
#9  0x0000000000f9b26e _start (scylla + 0xd9b26e)
Stack trace of thread 209113:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209115:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209107:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209118:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209096:
#0  0x0000000004097426 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97426)
#1  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#2  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209097:
#0  0x0000000004087a98 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3e87a98)
#1  0x00000000040b533b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3eb533b)
#2  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#3  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#4  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#5  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#6  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#7  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209102:
#0  0x0000000004087a98 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3e87a98)
#1  0x00000000040b533b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3eb533b)
#2  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#3  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#4  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#5  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#6  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#7  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209104:
#0  0x0000000004087a98 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3e87a98)
#1  0x00000000040b533b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3eb533b)
#2  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#3  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#4  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#5  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#6  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#7  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209093:
#0  0x0000000003ffd35b crc32_pclmul_avx (scylla + 0x3dfd35b)
#1  0x0000000003ffcec1 libdeflate_crc32 (scylla + 0x3dfcec1)
#2  0x00000000019d75c4 _ZN32compressed_file_data_source_implI11crc32_utilsE3getEv (scylla + 0x17d75c4)
#3  0x0000000001952a81 _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables2mx27data_consume_rows_context_mEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x1752a81)
#4  0x00000000019575ba _ZZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEvENKUlvE0_clEv (scylla + 0x17575ba)
#5  0x000000000194f8b8 _ZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEv (scylla + 0x174f8b8)
#6  0x00000000019301bc _ZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x17301bc)
#7  0x00000000019327dd _ZThn64_N8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x17327dd)
#8  0x00000000015c1db0 _ZN22mutation_reader_mergerclEv (scylla + 0x13c1db0)
#9  0x00000000015fb13d _ZN24mutation_fragment_mergerI22mutation_reader_mergerEclEv (scylla + 0x13fb13d)
#10 0x00000000015fa9d1 _ZN7seastar6repeatIZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvEUlvE_EENS_6futureIvEEOT_ (scylla + 0x13fa9d1)
#11 0x00000000015f9c57 _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x13f9c57)
#12 0x000000000169495f _ZN7seastar8futurizeINS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEE6invokeIRZN23flat_mutation_reader_v24impl16consume_pausableIZ15downgrade_to_v1S8_EN19transforming_reader8consumerEEENS1_IvEET_EUlvE_JEEES5_OSE_DpOT0_ (scylla + 0x149495f)
#13 0x00000000016916f1 _ZZ15downgrade_to_v123flat_mutation_reader_v2EN19transforming_reader11fill_bufferEv (scylla + 0x14916f1)
#14 0x0000000001a25a66 _ZZZN8sstables10compaction7consumeEvENUl20flat_mutation_readerE_clES1_ENUlvE_clEv (scylla + 0x1825a66)
#15 0x0000000001a25354 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction7consumeEvENUl20flat_mutation_readerE_clES7_EUlvE_JEEENS_8futurizeINSt13invoke_resultIT_JDpT0_EE4typeEE4typeENS_17thread_attributesEOSC_DpOSD_EUlvE_E4callEPKS2_ (scylla + 0x1825354)
#16 0x00000000042e78a2 _ZN7seastar14thread_context4mainEv (scylla + 0x40e78a2)
Stack trace of thread 209109:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209094:
#0  0x000000000147d769 _ZN8logalloc11region_impl4freeEPvm (scylla + 0x127d769)
#1  0x00000000016dfd8d _ZN5bplus4treeImN11cached_file11cached_pageENS1_24page_idx_less_comparatorELm12ELNS_10key_searchE0ELNS_10with_debugE0EE8iterator5eraseES3_ (scylla + 0x14dfd8d)
#2  0x00000000016df6c5 _ZN11cached_file11cached_page10on_evictedEv (scylla + 0x14df6c5)
#3  0x00000000013bdff3 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x11bdff3)
#4  0x00000000014509fe _ZN8logalloc11region_impl10evict_someEv (scylla + 0x12509fe)
#5  0x000000000145cb19 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x125cb19)
#6  0x0000000001458664 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x1258664)
#7  0x0000000001447fdc _ZN8logalloc12segment_pool16allocate_segmentEm (scylla + 0x1247fdc)
#8  0x00000000014667c8 _ZN8logalloc18allocating_section7reserveEv (scylla + 0x12667c8)
#9  0x00000000017c0ff1 _ZN8sstables27index_consume_entry_contextINS_14index_consumerEE13process_stateERN7seastar16temporary_bufferIcEE (scylla + 0x15c0ff1)
#10 0x00000000017bdabf _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables27index_consume_entry_contextINS4_14index_consumerEEEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x15bdabf)
#11 0x00000000017c60c1 _ZN7seastar8internal8repeaterIZNS_12input_streamIcE7consumeISt17reference_wrapperIN8sstables27index_consume_entry_contextINS6_14index_consumerEEEEEENS_6futureIvEEOT_EUlvE_E15run_and_disposeEv (scylla + 0x15c60c1)
#12 0x0000000004077f65 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3e77f65)
#13 0x0000000004079358 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79358)
#14 0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#15 0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#16 0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#17 0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209114:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209101:
#0  0x00007fe1419d0e10 LZ4_compress_fast_extState (liblz4.so.1 + 0x6e10)
#1  0x00007fe1419d188f LZ4_compress_fast (liblz4.so.1 + 0x788f)
#2  0x00000000016db686 _ZNK13lz4_processor8compressEPKcmPcm (scylla + 0x14db686)
#3  0x00000000019d8c17 _ZN30compressed_file_data_sink_implI11crc32_utilsL24compressed_checksum_mode1EE3putEN7seastar16temporary_bufferIcEE (scylla + 0x17d8c17)
#4  0x0000000001246226 _ZN7seastar13output_streamIcE10slow_writeEPKcm (scylla + 0x1046226)
#5  0x0000000001986c23 _ZN8sstables2mc6writer15write_clusteredERK14clustering_rowm (scylla + 0x1786c23)
#6  0x000000000198831c _ZN8sstables2mc6writer15write_clusteredI14clustering_rowEEvRKT_ (scylla + 0x178831c)
#7  0x0000000001987cb3 _ZN8sstables2mc6writer7consumeEO14clustering_row (scylla + 0x1787cb3)
#8  0x0000000001a3228c _ZN22compact_mutation_stateIL19emit_only_live_rows0EL20compact_for_sstables1EE7consumeIN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEN7seastar10bool_classINS7_18stop_iteration_tagEEEO14clustering_rowRT_RT0_ (scylla + 0x183228c)
#9  0x0000000001a30098 _ZNO17mutation_fragment7consumeIN20flat_mutation_reader4impl16consumer_adapterI35stable_flattened_mutations_consumerI22compact_for_compactionIN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEEEEEDcRT_ (scylla + 0x1830098)
#10 0x0000000001a25b7b _ZZZN8sstables10compaction7consumeEvENUl20flat_mutation_readerE_clES1_ENUlvE_clEv (scylla + 0x1825b7b)
#11 0x0000000001a25354 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction7consumeEvENUl20flat_mutation_readerE_clES7_EUlvE_JEEENS_8futurizeINSt13invoke_resultIT_JDpT0_EE4typeEE4typeENS_17thread_attributesEOSC_DpOSD_EUlvE_E4callEPKS2_ (scylla + 0x1825354)
#12 0x00000000042e78a2 _ZN7seastar14thread_context4mainEv (scylla + 0x40e78a2)
Stack trace of thread 209117:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209103:
#0  0x00000000040b5564 _ZN7seastar7reactor26io_queue_submission_pollfn4pollEv (scylla + 0x3eb5564)
#1  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#2  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#3  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#4  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#5  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#6  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209106:
#0  0x00007fe14117694c read (libpthread.so.0 + 0x1294c)
#1  0x00000000040b9df5 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3eb9df5)
#2  0x00000000040ba160 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3eba160)
#3  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#4  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#5  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)
Stack trace of thread 209092:
#0  0x0000000004087a12 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3e87a12)
#1  0x00000000040b533b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3eb533b)
#2  0x0000000004097429 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3e97429)
#3  0x0000000004079386 _ZN7seastar7reactor6do_runEv (scylla + 0x3e79386)
#4  0x00000000040982b6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_99E9_M_invokeERKSt9_Any_data (scylla + 0x3e982b6)
#5  0x000000000404a57b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3e4a57b)
#6  0x00007fe14116d299 start_thread (libpthread.so.0 + 0x9299)
#7  0x00007fe14067f353 __clone (libc.so.6 + 0x100353)

Download instructions

gsutil cp gs://upload.scylladb.com/core.scylla.113.ca68faa9ae854b6bb1c94e3741583354.209091.1640343486000000000000/core.scylla.113.ca68faa9ae854b6bb1c94e3741583354.209091.1640343486000000000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.113.ca68faa9ae854b6bb1c94e3741583354.209091.1640343486000000000000.gz

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor b282f96d-098a-4654-8560-81f5fee4f869
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs b282f96d-098a-4654-8560-81f5fee4f869

Test id: b282f96d-098a-4654-8560-81f5fee4f869

Logs:
grafana - https://cloudius-jenkins-test.s3.amazonaws.com/b282f96d-098a-4654-8560-81f5fee4f869/20211224_115115/grafana-screenshot-gemini-1tb-10h-scylla-per-server-metrics-nemesis-20211224_115331-gemini-1tb-10h-master-monitor-node-b282f96d-1.png
grafana - https://cloudius-jenkins-test.s3.amazonaws.com/b282f96d-098a-4654-8560-81f5fee4f869/20211224_115115/grafana-screenshot-overview-20211224_115115-gemini-1tb-10h-master-monitor-node-b282f96d-1.png
db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/b282f96d-098a-4654-8560-81f5fee4f869/20211224_115934/db-cluster-b282f96d.tar.gz
loader-set - https://cloudius-jenkins-test.s3.amazonaws.com/b282f96d-098a-4654-8560-81f5fee4f869/20211224_115934/loader-set-b282f96d.tar.gz
monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/b282f96d-098a-4654-8560-81f5fee4f869/20211224_115934/monitor-set-b282f96d.tar.gz
sct - https://cloudius-jenkins-test.s3.amazonaws.com/b282f96d-098a-4654-8560-81f5fee4f869/20211224_115934/sct-runner-b282f96d.tar.gz

Jenkins job URL

roydahan wrote this answer on 2022-01-02

Maybe a dup of #9821

avikivity wrote this answer on 2022-01-04

Closing as dup of #9821. Update your scylla.

More Details About Repo
Owner Name scylladb
Repo Name scylla
Full Name scylladb/scylla
Language C++
Created Date 2014-12-24
Updated Date 2022-07-24
Star Count 8131
Watcher Count 340
Fork Count 911
Issue Count 2172

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
Elixir - possible flaky test 0 2022-06-20 2022-09-08
FileUpload React component isnt working 1 2021-09-01 2022-09-22
95.0.4638.69-1.1 fails to compile on Ubuntu Impish with Clang 13 3 2021-10-30 2022-09-06
Warn user when OTP slot is overwritten 1 2018-09-04 2022-08-07
version numbers missing on macOS 8 2018-09-12 2022-09-21
Version 5.2 (149) crashing after some time: TigaseSQLite3.DBError.sqliteError(errorCode: 1, message: Optional("near \"(\": syntax error")) 3 2022-03-21 2022-08-09
Issue: Media URLs are inserted relative to repo root instead of the public directory 4 2022-06-12 2022-09-16
DvorakJPローマ字テーブルについて 0 2020-07-31 2022-09-24
关于docker的使用求助 8 2021-02-14 2022-01-15
thats what i get when i call openAudioSession()i() [BUG]: 2 2022-04-07 2022-09-13
Hamachi cannot connect becuase of port forwarding. 1 2022-04-04 2022-09-02
is there a possibility to drag divs within another draggable div 1 2017-08-24 2022-01-20
Split authors field in plugin manifest 0 2022-02-17 2022-09-22
LevelZero Driver-API Versions 6 2021-11-04 2022-09-24
[Win10Pro.2004] COM exception when executing patchfluent 0 2021-11-30 2022-09-12
Issue: Getting NextPaginationToken as null for reviews api 0 2021-06-06 2022-09-11
Editable Profile Image Feature 1 2021-03-21 2022-08-14
Auto reload bib file in node.js 2 2022-02-01 2022-09-23
The BOM is readed as a value in the content 4 2022-06-09 2022-08-26
Сhanging GEOSERVER_DATA_DIR breaks the container 2 2022-07-29 2022-08-20
Clarify stance on encryption export requirements for App Store 1 2021-03-02 2022-09-11
[doc] Distinction between Guides and Examples 0 2022-05-10 2022-06-09
Fix Gateway API conformance test HTTPRouteHostnameIntersection 0 2022-06-30 2022-09-12
http-terminator: Please verify that the package.json has a valid "main" entry. 2 2021-09-17 2022-08-26
Seeming Grace Period Shutdown Default Mismatch 0 2021-08-25 2022-08-26
Cannot use Dokany on ARM64 8 2021-12-22 2022-09-14
Termux: if install via install script, use yarn 8 2021-10-10 2022-09-11
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration sentry.0201_semver_package is applied before its dependency sentry.0001_squashed_0200_release_indices on database 'default' 1 2021-07-16 2022-09-14
profile identicons missing in invite (possible duplicate) 0 2022-09-21 2022-09-18
Action buttons for accepting/rejecting contact requests should be identical in different places 0 2022-09-21 2022-09-18
pending requests initially missing in community settings immediatly after invites are sent (possible duplicate) 2 2022-09-21 2022-09-18
Contact request: the text of contact request is shown as push notification when contact request is accepted 0 2022-09-21 2022-09-18
Depfu Error: No dependency files found 0 2021-12-23 2022-07-03
`ProceduralSkyMaterial` uses all `DirectionalLight3D` in the scene as suns 6 2021-10-19 2022-08-26
Make providers flyweight 0 2022-07-02 2022-09-20
amdgpu: black screen after resume from suspend 1 2022-04-26 2022-09-22
No Blocklists page available 2 2022-03-28 2022-09-23
MicroProfile 4.1 Compatible Certification Request - Payara Server Enterprise 5.43.0 0 2022-09-07 2022-09-17
Please increase max wheel size limit for intel-extionsion-for-pytorch projects(test and prod) to 250MB. 2 2022-02-15 2022-09-19
🤖 Sync error for commit fbfa46 0 2021-09-30 2022-09-27
the trait bound `OsRng: ecdsa::elliptic_curve::rand_core::RngCore` is not satisfied 3 2021-04-07 2022-07-26
[Test Proxy] Failure when passing in one client to multiple AvroEncoder objects 0 2022-04-22 2022-09-24
Migrate third party container registries to first party 0 2022-04-22 2022-09-20
`GradleProperties has not been loaded yet` error when re-using stored configuration cache entry 1 2021-11-24 2022-07-17
Not working: RPKI validator 6 2021-10-01 2022-08-27
Parallelize Type Deduction 1 2021-02-05 2022-07-29
Unexpected Panic While Using Struct Update Syntax On Cloned Struct Reference 1 2022-04-28 2022-09-23
azuread_service_principal_delegated_permission_grant when a grant already exists 5 2021-11-26 2022-09-22
Triton int64 overflow bug 2 2022-03-15 2022-09-26
[flake] AsyncTraceMethodInstrumentationTest.testWithCrossedThreshold_Generic 0 2022-04-21 2022-09-04