[SON-107] Merge develop branch to SONs-base (#166)

* fix rng and get_winner_numbers implemented

* coipied code for bitshares fixing 429 and 433 isuues

* ticket_purchase_operation implemented. added lottery_options to asset

* lottery end implemented

* minor logic changes. added db_api and cli_wallet methods

* fix reindex on peerplays network

* fix some tests. add gitlab-ci.yml

* add pull to gitlab-ci

* fix

* fix and comment some tests

* added owner to lottery_asset_options. commented async call in on_applied_block callback

* added get_account_lotteries method to db_api and cli, lottery end_date and ticket_price verification

* merge get_account_lotteries branch. fix create_witness test

* fix test genesis and end_date verification

* fixed indices sorting and lottery end checking by date

* update db_version for replay and removed duplicate include files

* Added ntp and upgraded boost version

* Revert "GPOS protocol"

* need to remove backup files

* virtual-op-fix for deterministic virtual_op number

* Merged beatrice into 5050

* Updated gitmodules, changes to allow voting on lottery fee

* Removed submodule libraries/fc

* Added libraries/fc

* added missing , in types.hpp

* Added sweeps parameters to parameter_extension

* added missing comma in operations.hpp, small changes to config.hpp

* fixed returntype in chain_parameters.hpp

* removed sweeps_parameter_extensions

* Changed fc library

* fixed asset_object

* Changed peerplays-fc submodule

* Changed fc submodule to ubuntu 18.04 upgrade

* Removed submodule libraries/fc

* Added fc library back

* fix casting in overloaded function

* Removed blind_sign and unblind_signature functions

* Added new lottery_asset_create_operation

* Changed sweeps hardfork time

* Removed redundant if from asset_evaluator and fixed db_notify

* fixed duplicate code in fee_tests

* removed redundant tgenesis file

* Enable building on Ubuntu 18.04 using GCC 7 compiler

* fix: is_benefactor_reward had the default value of true when not set

* Docker file for Ubuntu 18.04

Base image updated to Unbuntu 18.04
Prerequisite list updated
Basic configuration updated

* Quick fix: Added missing package pkg-config

* Docker file updates

* 5050 fee update and compilation error fix

* Dockerfile, set system locale

Prevents locale::facet::_S_create_c_locale name error

* Update README.md

Fix typo

* Update README.md

* Changed hardfork time for SWEEPS and Core-429

* revert master changes that were brought in previous commit

* Fixed error when account_history_object with id 0 doesnt exist

* Fixed error while loading object database

* test for zero id object in account history

* Reorder operations in Dockerfile, to make image creation faster

- Reorder prevents unnecessary building of Boost libraries

* Fix for irrelevant signature included issue

* fix copyrigth messages order

* remove double empty lines

* Backport fix for `get_account_history` from https://github.com/bitshares/bitshares-core/pull/628 and add additional account history test case

* NTP client back

* GRPH-53-Log_format_error

* Merge pull request #1036 from jmjatlanta/issue_730

Add fail_reason to proposal_object

* Unit test case fixes and prepared SONs base

* Use offsetof instead of custom macro

* Hide some compiler warnings

* Make all the tests compile

* Add nullptr check in api.cpp for easier testing

* Add test case for broadcast_trx_with_callback API

* Unit test case fixes and prepared SONs base

* Merge pull request #714 from pmconrad/json_fix

JSON fix

* Increase max depth for trx confirmation callback

* Adapt to variant API with `max_depth` argument

* Update fc submodule

* Created unit test for #325

* remove needless find()

* GRPH-4-CliWallet_crash_ctrlD

* fix copyright message

* Make all the tests compile

* increase delay for node connection

* Increase block creation timeout to 2500ms

* remove cache from cli get_account

* add cli tests framework

* Adjust newly merged code to new API

* Improved resilience of block database against corruption

* Merged changes from Bitshares PR 1036

* GRPH-76 - Short-cut long sequences of missed blocks

Fixes database::update_global_dynamic_data to speed up counting missed blocks.
(This also fixes a minor issue with counting - the previous algorithm would skip missed blocks for the witness who signed the first block after the gap.)

* Moved reindex logic into database / chain_database, make use of additional blocks in block_database

Fixed tests wrt db.open

* Enable undo + fork database for final blocks in a replay

Dont remove blocks from block db when popping blocks, handle edge case in replay wrt fork_db, adapted unit tests

* Log starting block number of replay

* Prevent unsigned integer underflow

* Fixed lock detection

* Dont leave _data_dir empty if db is locked

* Writing the object_database is now almost atomic

* Improved consistency check for block_log

* Cut back block_log index file if inconsistent

* Fixed undo_database

* Added test case for broken merge on empty undo_db

* Merge pull request #938 from bitshares/fix-block-storing

Store correct block ID when switching forks

* exclude second undo_db.enable() call in some cases

* Add missing change

* change bitshares to core in message

* Fixed integer overflow issue

* Fix for for history ID mismatch ( Bitshares PR #875 )

* Update the FC submodule with the changes for GRPH-4

* Fix #436 object_database created outside of witness data directory

* supplement more comments on database::_opened variable

* prevent segfault when destructing application obj

* Fixed duplicate ops returned from get_account_history

* minor performance improvement

* Added comment

* Merged Bitshares PR #1462 and compilation fixes

* Support/gitlab (#123)

* Updated gitlab process

* Fix undefined references in cli test

* Fixed test failures and compilation issue

* Fixed account_history_pagination test

* Fix compilation in debug mode

* Removed unrelated comment

* Skip auth check when pushing self-generated blocks

* Extract public keys before pushing a transaction

* Dereference chain_database shared_ptr

* Updated transaction::signees to mutable

and
* updated get_signature_keys() to return a const reference,
* get_signature_keys() will update signees on first call,
* modified test cases and wallet.cpp accordingly,
* no longer construct a new signed_transaction object before pushing

* Added get_asset_count API

* Allow sufficient space for new undo_session

* Throw for deep nesting

* No longer extract public keys before pushing a trx

and removed unused new added constructor and _get_signature_keys() function from signed_transaction struct

* Added cli_test to CI

* use random port numbers in app_test (#154)

* proposal fail_reason bug fixed (#157)

* Added Sonarcloud code_quality to CI (#159)

* Added sonarcloud analysis (#158)

* fix for lottery end

* fix declarations

* fix declarations

* fix boost integer

* fix compilation

* fix chain tests

* fix app_test

* try to fix cli test

* fix incorrect max_depth param

* working cli test

* correct fc version

.dockerignore

@@ -1,2 +1 @@
 build
-

.gitignore

@@ -9,6 +9,8 @@ compile_commands.json
 moc_*
 *.moc
 hardfork.hpp
+build_xc
+data
 
 libraries/utilities/git_revision.cpp
 

.gitlab-ci.yml

@@ -0,0 +1,32 @@
+include:
+  - template: Code-Quality.gitlab-ci.yml
+
+stages:
+  - build
+  - test
+
+build:
+  stage: build
+  script:
+    - git submodule update --init --recursive
+    - cmake .
+    - make -j$(nproc)
+  artifacts:
+    untracked: true
+    paths:
+      - libraries/
+      - programs/
+      - tests/
+  tags:
+    - builder
+    
+test:
+  stage: test
+  dependencies: 
+    - build
+  script:
+    - ./tests/betting_test
+    - ./tests/chain_test
+    - ./tests/cli_test
+  tags:
+    - builder

.gitmodules

@@ -4,6 +4,4 @@
     ignore = dirty
 [submodule "libraries/fc"]
 	path = libraries/fc
-    url = https://github.com/PBSA/peerplays-fc.git
+	url = https://github.com/PBSA/peerplays-fc
-    branch = feature/ubuntu18.04-upgrade
-    ignore = dirty

CMakeLists.txt

@@ -135,7 +135,7 @@ else( WIN32 ) # Apple AND Linux
     endif( APPLE )
 
     if( "${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" )
-        set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-builtin-memcmp" )
+        set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-builtin-memcmp -Wno-class-memaccess -Wno-parentheses -Wno-terminate -Wno-invalid-offsetof" )
     elseif( "${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang" )
         if( CMAKE_CXX_COMPILER_VERSION VERSION_EQUAL 4.0.0 OR CMAKE_CXX_COMPILER_VERSION VERSION_GREATER 4.0.0 )
             set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-invalid-partial-specialization" )

Dockerfile

@@ -25,6 +25,10 @@ RUN \
       locales \
       ntp \
       pkg-config \
+      ntp \
+      pkg-config \
+      doxygen \
+      ca-certificates \
       wget \
     && \
     apt-get clean && \

libraries/app/api.cpp

@@ -160,7 +160,10 @@ namespace graphene { namespace app {
              {
                 auto block_num = b.block_num();
                 auto& callback = _callbacks.find(id)->second;
-                fc::async( [capture_this,this,id,block_num,trx_num,trx,callback](){ callback( fc::variant(transaction_confirmation{ id, block_num, trx_num, trx}) ); } );
+                fc::async( [capture_this,this,id,block_num,trx_num,trx,callback]() {
+                   callback( fc::variant( transaction_confirmation{ id, block_num, trx_num, trx },
+                                          GRAPHENE_MAX_NESTED_OBJECTS ) );
+                } );
              }
           }
        }
@@ -189,6 +192,7 @@ namespace graphene { namespace app {
     void network_broadcast_api::broadcast_block( const signed_block& b )
     {
        _app.chain_database()->push_block(b);
+       if( _app.p2p_node() != nullptr )
           _app.p2p_node()->broadcast( net::block_message( b ));
     }
 
@@ -197,6 +201,7 @@ namespace graphene { namespace app {
        trx.validate();
        _callbacks[trx.id()] = cb;
        _app.chain_database()->push_transaction(trx);
+       if( _app.p2p_node() != nullptr )
             _app.p2p_node()->broadcast_transaction(trx);
     }
 
@@ -212,7 +217,7 @@ namespace graphene { namespace app {
 
             if (_on_pending_transaction)
             {
-                _on_pending_transaction(fc::variant(transaction));
+                _on_pending_transaction(fc::variant(transaction, GRAPHENE_MAX_NESTED_OBJECTS));
             }
         });
 
@@ -554,19 +559,25 @@ namespace graphene { namespace app {
         const auto& db = *_app.chain_database();
         FC_ASSERT( limit <= 100 );
         vector<operation_history_object> result;
-       const auto& stats = account(db).statistics(db);
+        try {
-       if( stats.most_recent_op == account_transaction_history_id_type() ) return result;
+           const account_transaction_history_object& node = account(db).statistics(db).most_recent_op(db);
-       const account_transaction_history_object* node = &stats.most_recent_op(db);
+           if(start == operation_history_id_type() || start.instance.value > node.operation_id.instance.value)
-       if( start == operation_history_id_type() )
+              start = node.operation_id;
-          start = node->operation_id;
+        } catch(...) { return result; }
 
-       while(node && node->operation_id.instance.value > stop.instance.value && result.size() < limit)
+        const auto& hist_idx = db.get_index_type<account_transaction_history_index>();
+        const auto& by_op_idx = hist_idx.indices().get<by_op>();
+        auto index_start = by_op_idx.begin();
+        auto itr = by_op_idx.lower_bound(boost::make_tuple(account, start));
+
+        while(itr != index_start && itr->account == account && itr->operation_id.instance.value > stop.instance.value && result.size() < limit)
         {
-          if( node->operation_id.instance.value <= start.instance.value )
+           if(itr->operation_id.instance.value <= start.instance.value)
-             result.push_back( node->operation_id(db) );
+              result.push_back(itr->operation_id(db));
-          if( node->next == account_transaction_history_id_type() )
+           --itr;
-             node = nullptr;
+        }
-          else node = &node->next(db);
+        if(stop.instance.value == 0 && result.size() < limit && itr->account == account) {
+          result.push_back(itr->operation_id(db));
         }
 
         return result;
@@ -599,6 +610,11 @@ namespace graphene { namespace app {
              node = nullptr;
           else node = &node->next(db);
        }
+       if( stop.instance.value == 0 && result.size() < limit ) {
+          auto head = db.find(account_transaction_history_id_type());
+          if (head != nullptr && head->account == account && head->operation_id(db).op.which() == operation_id)
+            result.push_back(head->operation_id(db));
+       }
        return result;
     }
 

libraries/app/application.cpp

@@ -142,7 +142,7 @@ namespace detail {
          if( _options->count("seed-nodes") )
          {
             auto seeds_str = _options->at("seed-nodes").as<string>();
-            auto seeds = fc::json::from_string(seeds_str).as<vector<string>>();
+            auto seeds = fc::json::from_string(seeds_str).as<vector<string>>(2);
             for( const string& endpoint_string : seeds )
             {
                try {
@@ -166,6 +166,24 @@ namespace detail {
                 "159.69.223.206:7777",
                 "51.38.237.243:9666",
                 "pbsa-beatrice.blockchainprojectsbv.com:9195"
+              // OTTHER SEEDS:
+              //  "seed.ppy.blckchnd.com:6112",       // blckchnd 
+              //  "ppy.esteem.ws:7777",               // good-karma
+              //  "peerplays.bitcoiner.me:9777",      // bitcoiner
+              //  "peerplays.roelandp.nl:9777",       // roelandp
+              //  "78.46.95.153:7777",                // theprophet0 
+              //  "ppyseed.bacchist.me:42420",        // bacchist-witness
+              //  "5.9.18.213:18828",                 // pfunk
+              //  "31.171.244.121:7777",              // taconator
+              //  "seed.peerplaysdb.com:9777",        // jesta
+              //  "ppy-seed.xeldal.com:19777",        // xeldal
+              //  "peerplays-seed.altcap.io:61388",   // winner.winner.chicken.dinner
+              //  "seed.peerplaysnodes.com:9777",     // wackou
+              //  "peerplays-seed.privex.io:7777",    // someguy123/privex
+              //  "51.15.78.16:9777",	               // agoric.systems
+              //  "212.71.253.163:9777",	            // xtar
+              //  "51.15.35.96:9777",	               // lafona
+              //  "anyx.ca:9777"                      // anyx
             };
 
             for( const string& endpoint_string : seeds )
@@ -226,7 +244,7 @@ namespace detail {
 
       void new_connection( const fc::http::websocket_connection_ptr& c )
       {
-         auto wsc = std::make_shared<fc::rpc::websocket_api_connection>(*c);
+         auto wsc = std::make_shared<fc::rpc::websocket_api_connection>(c, GRAPHENE_MAX_NESTED_OBJECTS);
          auto login = std::make_shared<graphene::app::login_api>( std::ref(*_self) );
          login->enable_api("database_api");
 
@@ -292,7 +310,7 @@ namespace detail {
          _websocket_tls_server->start_accept();
       } FC_CAPTURE_AND_RETHROW() }
 
-      application_impl(application* self)
+      explicit application_impl(application* self)
          : _self(self),
            _chain_db(std::make_shared<chain::database>())
       {
@@ -300,7 +318,6 @@ namespace detail {
 
       ~application_impl()
       {
-         fc::remove_all(_data_dir / "blockchain/dblock");
       }
 
       void set_dbg_init_key( genesis_state_type& genesis, const std::string& init_key )
@@ -309,21 +326,19 @@ namespace detail {
          public_key_type init_pubkey( init_key );
          for( uint64_t i=0; i<genesis.initial_active_witnesses; i++ )
             genesis.initial_witness_candidates[i].block_signing_key = init_pubkey;
-         return;
       }
 
       void startup()
       { try {
-         bool clean = !fc::exists(_data_dir / "blockchain/dblock");
+         fc::create_directories(_data_dir / "blockchain");
-         fc::create_directories(_data_dir / "blockchain/dblock");
 
-         auto initial_state = [&] {
+         auto initial_state = [this] {
             ilog("Initializing database...");
             if( _options->count("genesis-json") )
             {
                std::string genesis_str;
                fc::read_file_contents( _options->at("genesis-json").as<boost::filesystem::path>(), genesis_str );
-               genesis_state_type genesis = fc::json::from_string( genesis_str ).as<genesis_state_type>();
+               genesis_state_type genesis = fc::json::from_string( genesis_str ).as<genesis_state_type>( 20 );
                bool modified_genesis = false;
                if( _options->count("genesis-timestamp") )
                {
@@ -356,7 +371,7 @@ namespace detail {
                graphene::egenesis::compute_egenesis_json( egenesis_json );
                FC_ASSERT( egenesis_json != "" );
                FC_ASSERT( graphene::egenesis::get_egenesis_json_hash() == fc::sha256::hash( egenesis_json ) );
-               auto genesis = fc::json::from_string( egenesis_json ).as<genesis_state_type>();
+               auto genesis = fc::json::from_string( egenesis_json ).as<genesis_state_type>( 20 );
                genesis.initial_chain_id = fc::sha256::hash( egenesis_json );
                return genesis;
             }
@@ -372,7 +387,7 @@ namespace detail {
             loaded_checkpoints.reserve( cps.size() );
             for( auto cp : cps )
             {
-               auto item = fc::json::from_string(cp).as<std::pair<uint32_t,block_id_type> >();
+               auto item = fc::json::from_string(cp).as<std::pair<uint32_t,block_id_type> >( 2 );
                loaded_checkpoints[item.first] = item.second;
             }
          }
@@ -381,64 +396,17 @@ namespace detail {
          bool replay = false;
          std::string replay_reason = "reason not provided";
 
-         // never replay if data dir is empty
-         if( fc::exists( _data_dir ) && fc::directory_iterator( _data_dir ) != fc::directory_iterator() )
-         {
          if( _options->count("replay-blockchain") )
-            {
+            _chain_db->wipe( _data_dir / "blockchain", false );
-               replay = true;
-               replay_reason = "replay-blockchain argument specified";
-            }
-            else if( !clean )
-            {
-               replay = true;
-               replay_reason = "unclean shutdown detected";
-            }
-            else if( !fc::exists( _data_dir / "db_version" ) )
-            {
-               replay = true;
-               replay_reason = "db_version file not found";
-            }
-            else
-            {
-               std::string version_string;
-               fc::read_file_contents( _data_dir / "db_version", version_string );
 
-               if( version_string != GRAPHENE_CURRENT_DB_VERSION )
-               {
-                   replay = true;
-                   replay_reason = "db_version file content mismatch";
-               }
-            }
-         }
-
-         if( !replay )
-         {
          try
          {
-               _chain_db->open( _data_dir / "blockchain", initial_state );
+            _chain_db->open( _data_dir / "blockchain", initial_state, GRAPHENE_CURRENT_DB_VERSION );
          }
          catch( const fc::exception& e )
          {
-               ilog( "Caught exception ${e} in open()", ("e", e.to_detail_string()) );
+            elog( "Caught exception ${e} in open(), you might want to force a replay", ("e", e.to_detail_string()) );
-
+            throw;
-               replay = true;
-               replay_reason = "exception in open()";
-            }
-         }
-
-         if( replay )
-         {
-            ilog( "Replaying blockchain due to: ${reason}", ("reason", replay_reason) );
-
-            fc::remove_all( _data_dir / "db_version" );
-            _chain_db->reindex( _data_dir / "blockchain", initial_state() );
-
-            const auto mode = std::ios::out | std::ios::binary | std::ios::trunc;
-            std::ofstream db_version( (_data_dir / "db_version").generic_string().c_str(), mode );
-            std::string version_string = GRAPHENE_CURRENT_DB_VERSION;
-            db_version.write( version_string.c_str(), version_string.size() );
-            db_version.close();
          }
 
          if( _options->count("force-validate") )
@@ -447,9 +415,21 @@ namespace detail {
             _force_validate = true;
          }
 
-         if( _options->count("api-access") )
+         if( _options->count("api-access") ) {
-            _apiaccess = fc::json::from_file( _options->at("api-access").as<boost::filesystem::path>() )
+
-               .as<api_access>();
+            if(fc::exists(_options->at("api-access").as<boost::filesystem::path>()))
+            {
+               _apiaccess = fc::json::from_file( _options->at("api-access").as<boost::filesystem::path>() ).as<api_access>( 20 );
+               ilog( "Using api access file from ${path}",
+                     ("path", _options->at("api-access").as<boost::filesystem::path>().string()) );
+            }
+            else
+            {
+               elog("Failed to load file from ${path}",
+                  ("path", _options->at("api-access").as<boost::filesystem::path>().string()));
+               std::exit(EXIT_FAILURE);
+            }
+         }
          else
          {
             // TODO:  Remove this generous default access policy
@@ -992,7 +972,7 @@ void application::initialize(const fc::path& data_dir, const boost::program_opti
       if( fc::exists(genesis_out) )
       {
          try {
-            genesis_state = fc::json::from_file(genesis_out).as<genesis_state_type>();
+            genesis_state = fc::json::from_file(genesis_out).as<genesis_state_type>( 20 );
          } catch(const fc::exception& e) {
             std::cerr << "Unable to parse existing genesis file:\n" << e.to_string()
                       << "\nWould you like to replace it? [y/N] ";

libraries/app/config_util.cpp

@@ -31,6 +31,7 @@
 #include <fc/log/console_appender.hpp>
 #include <fc/log/file_appender.hpp>
 #include <fc/log/logger_config.hpp>
+#include <fc/log/logger.hpp>
 
 #include <boost/property_tree/ptree.hpp>
 #include <boost/property_tree/ini_parser.hpp>
@@ -150,8 +151,8 @@ static fc::optional<fc::logging_config> load_logging_config_from_ini_file(const
             console_appender_config.level_colors.emplace_back(
             fc::console_appender::level_color(fc::log_level::error,
                                               fc::console_appender::color::cyan));
-            console_appender_config.stream = fc::variant(stream_name).as<fc::console_appender::stream::type>();
+            console_appender_config.stream = fc::variant(stream_name).as<fc::console_appender::stream::type>(GRAPHENE_MAX_NESTED_OBJECTS);
-            logging_config.appenders.push_back(fc::appender_config(console_appender_name, "console", fc::variant(console_appender_config)));            
+            logging_config.appenders.push_back(fc::appender_config(console_appender_name, "console", fc::variant(console_appender_config, 1)));
             found_logging_config = true;
          }
          else if (boost::starts_with(section_name, file_appender_section_prefix))
@@ -172,7 +173,7 @@ static fc::optional<fc::logging_config> load_logging_config_from_ini_file(const
             file_appender_config.rotate = true;
             file_appender_config.rotation_interval = fc::minutes(interval);
             file_appender_config.rotation_limit = fc::days(limit);
-            logging_config.appenders.push_back(fc::appender_config(file_appender_name, "file", fc::variant(file_appender_config)));
+            logging_config.appenders.push_back(fc::appender_config(file_appender_name, "file", fc::variant(file_appender_config, 1)));
             found_logging_config = true;
          }
          else if (boost::starts_with(section_name, logger_section_prefix))
@@ -181,7 +182,7 @@ static fc::optional<fc::logging_config> load_logging_config_from_ini_file(const
             std::string level_string = section_tree.get<std::string>("level");
             std::string appenders_string = section_tree.get<std::string>("appenders");
             fc::logger_config logger_config(logger_name);
-            logger_config.level = fc::variant(level_string).as<fc::log_level>();
+            logger_config.level = fc::variant(level_string).as<fc::log_level>(1);
             boost::split(logger_config.appenders, appenders_string,
                          boost::is_any_of(" ,"),
                          boost::token_compress_on);

libraries/app/database_api.cpp

@@ -47,9 +47,6 @@ typedef std::map< std::pair<graphene::chain::asset_id_type, graphene::chain::ass
 
 namespace graphene { namespace app {
 
-class database_api_impl;
-
-
 class database_api_impl : public std::enable_shared_from_this<database_api_impl>
 {
    public:
@@ -103,6 +100,7 @@ class database_api_impl : public std::enable_shared_from_this<database_api_impl>
       vector<optional<asset_object>> get_assets(const vector<asset_id_type>& asset_ids)const;
       vector<asset_object>           list_assets(const string& lower_bound_symbol, uint32_t limit)const;
       vector<optional<asset_object>> lookup_asset_symbols(const vector<string>& symbols_or_ids)const;
+      uint64_t                       get_asset_count()const;
 
       // Peerplays
       vector<sport_object>                list_sports() const; 
@@ -113,6 +111,18 @@ class database_api_impl : public std::enable_shared_from_this<database_api_impl>
       vector<bet_object> get_unmatched_bets_for_bettor(betting_market_id_type, account_id_type) const;
       vector<bet_object> get_all_unmatched_bets_for_bettor(account_id_type) const;
 
+      // Lottery Assets
+      vector<asset_object> get_lotteries( asset_id_type stop  = asset_id_type(),
+                                          unsigned limit = 100,
+                                          asset_id_type start = asset_id_type() )const;
+      vector<asset_object> get_account_lotteries( account_id_type issuer, 
+                                                  asset_id_type stop,
+                                                  unsigned limit,
+                                                  asset_id_type start )const;
+      asset get_lottery_balance( asset_id_type lottery_id )const;
+      sweeps_vesting_balance_object get_sweeps_vesting_balance_object( account_id_type account )const;
+      asset get_sweeps_vesting_balance_available_for_claim( account_id_type account )const;
+   
       // Markets / feeds
       vector<limit_order_object>         get_limit_orders(asset_id_type a, asset_id_type b, uint32_t limit)const;
       vector<call_order_object>          get_call_orders(asset_id_type a, uint32_t limit)const;
@@ -161,8 +171,6 @@ class database_api_impl : public std::enable_shared_from_this<database_api_impl>
       vector<tournament_object> get_tournaments_by_state(tournament_id_type stop, unsigned limit, tournament_id_type start, tournament_state state);
       vector<tournament_id_type> get_registered_tournaments(account_id_type account_filter, uint32_t limit) const;
 
-      // gpos
-      gpos_info get_gpos_info(const account_id_type account) const;
 
    //private:
       template<typename T>
@@ -174,7 +182,6 @@ class database_api_impl : public std::enable_shared_from_this<database_api_impl>
 
          if( !is_subscribed_to_item(i) )
          {
-            idump((i));
             _subscribe_filter.insert( vec.data(), vec.size() );//(vecconst char*)&i, sizeof(i) );
          }
       }
@@ -208,7 +215,7 @@ class database_api_impl : public std::enable_shared_from_this<database_api_impl>
 
          auto sub = _market_subscriptions.find( market );
          if( sub != _market_subscriptions.end() ) {
-            queue[market].emplace_back( full_object ? obj->to_variant() : fc::variant(obj->id) );
+            queue[market].emplace_back( full_object ? obj->to_variant() : fc::variant(obj->id, 1) );
          }
       }
 
@@ -264,7 +271,7 @@ database_api_impl::database_api_impl( graphene::chain::database& db ):_db(db)
    _applied_block_connection = _db.applied_block.connect([this](const signed_block&){ on_applied_block(); });
 
    _pending_trx_connection = _db.on_pending_transaction.connect([this](const signed_transaction& trx ){
-                         if( _pending_trx_callback ) _pending_trx_callback( fc::variant(trx) );
+                         if( _pending_trx_callback ) _pending_trx_callback( fc::variant(trx, GRAPHENE_MAX_NESTED_OBJECTS) );
                       });
 }
 
@@ -515,6 +522,11 @@ vector<vector<account_id_type>> database_api::get_key_references( vector<public_
 vector<vector<account_id_type>> database_api_impl::get_key_references( vector<public_key_type> keys )const
 {
    wdump( (keys) );
+   
+   const auto& idx = _db.get_index_type<account_index>();
+   const auto& aidx = dynamic_cast<const base_primary_index&>(idx);
+   const auto& refs = aidx.get_secondary_index<graphene::chain::account_member_index>();
+   
    vector< vector<account_id_type> > final_result;
    final_result.reserve(keys.size());
 
@@ -534,10 +546,6 @@ vector<vector<account_id_type>> database_api_impl::get_key_references( vector<pu
       subscribe_to_item( a4 );
       subscribe_to_item( a5 );
 
-      const auto& idx = _db.get_index_type<account_index>();
-      const auto& aidx = dynamic_cast<const primary_index<account_index>&>(idx);
-      const auto& refs = aidx.get_secondary_index<graphene::chain::account_member_index>();
-      auto itr = refs.account_to_key_memberships.find(key);
       vector<account_id_type> result;
 
       for( auto& a : {a1,a2,a3,a4,a5} )
@@ -545,7 +553,7 @@ vector<vector<account_id_type>> database_api_impl::get_key_references( vector<pu
           auto itr = refs.account_to_address_memberships.find(a);
           if( itr != refs.account_to_address_memberships.end() )
           {
-             result.reserve( itr->second.size() );
+             result.reserve( result.size() + itr->second.size() );
              for( auto item : itr->second )
              {
                 wdump((a)(item)(item(_db).name));
@@ -554,9 +562,10 @@ vector<vector<account_id_type>> database_api_impl::get_key_references( vector<pu
           }
       }
 
+      auto itr = refs.account_to_key_memberships.find(key);
       if( itr != refs.account_to_key_memberships.end() )
       {
-         result.reserve( itr->second.size() );
+         result.reserve( result.size() + itr->second.size() );
          for( auto item : itr->second ) result.push_back(item);
       }
       final_result.emplace_back( std::move(result) );
@@ -589,7 +598,7 @@ bool database_api_impl::is_public_key_registered(string public_key) const
         return false;
     }
     const auto& idx = _db.get_index_type<account_index>();
-    const auto& aidx = dynamic_cast<const primary_index<account_index>&>(idx);
+    const auto& aidx = dynamic_cast<const base_primary_index&>(idx);
     const auto& refs = aidx.get_secondary_index<graphene::chain::account_member_index>();
     auto itr = refs.account_to_key_memberships.find(key);
     bool is_known = itr != refs.account_to_key_memberships.end();
@@ -630,14 +639,17 @@ std::map<string,full_account> database_api::get_full_accounts( const vector<stri
 
 std::map<std::string, full_account> database_api_impl::get_full_accounts( const vector<std::string>& names_or_ids, bool subscribe)
 {
-   idump((names_or_ids));
+   const auto& proposal_idx = _db.get_index_type<proposal_index>();
+   const auto& pidx = dynamic_cast<const base_primary_index&>(proposal_idx);
+   const auto& proposals_by_account = pidx.get_secondary_index<graphene::chain::required_approval_index>();
+   
    std::map<std::string, full_account> results;
 
    for (const std::string& account_name_or_id : names_or_ids)
    {
       const account_object* account = nullptr;
       if (std::isdigit(account_name_or_id[0]))
-         account = _db.find(fc::variant(account_name_or_id).as<account_id_type>());
+         account = _db.find(fc::variant(account_name_or_id, 1).as<account_id_type>(1));
       else
       {
          const auto& idx = _db.get_index_type<account_index>().indices().get<by_name>();
@@ -655,7 +667,6 @@ std::map<std::string, full_account> database_api_impl::get_full_accounts( const
          subscribe_to_item( account->id );
       }
 
-      // fc::mutable_variant_object full_account;
       full_account acnt;
       acnt.account = *account;
       acnt.statistics = account->statistics(_db);
@@ -664,20 +675,11 @@ std::map<std::string, full_account> database_api_impl::get_full_accounts( const
       acnt.lifetime_referrer_name = account->lifetime_referrer(_db).name;
       acnt.votes = lookup_vote_ids( vector<vote_id_type>(account->options.votes.begin(),account->options.votes.end()) );
 
-      // Add the account itself, its statistics object, cashback balance, and referral account names
-      /*
-      full_account("account", *account)("statistics", account->statistics(_db))
-            ("registrar_name", account->registrar(_db).name)("referrer_name", account->referrer(_db).name)
-            ("lifetime_referrer_name", account->lifetime_referrer(_db).name);
-            */
       if (account->cashback_vb)
       {
          acnt.cashback_balance = account->cashback_balance(_db);
       }
       // Add the account's proposals
-      const auto& proposal_idx = _db.get_index_type<proposal_index>();
-      const auto& pidx = dynamic_cast<const primary_index<proposal_index>&>(proposal_idx);
-      const auto& proposals_by_account = pidx.get_secondary_index<graphene::chain::required_approval_index>();
       auto  required_approvals_itr = proposals_by_account._account_to_proposals.find( account->id );
       if( required_approvals_itr != proposals_by_account._account_to_proposals.end() )
       {
@@ -688,12 +690,9 @@ std::map<std::string, full_account> database_api_impl::get_full_accounts( const
 
 
       // Add the account's balances
-      auto balance_range = _db.get_index_type<account_balance_index>().indices().get<by_account_asset>().equal_range(boost::make_tuple(account->id));
+      const auto& balances = _db.get_index_type< primary_index< account_balance_index > >().get_secondary_index< balances_by_account_index >().get_account_balances( account->id );
-      //vector<account_balance_object> balances;
+      for( const auto balance : balances )
-      std::for_each(balance_range.first, balance_range.second,
+         acnt.balances.emplace_back( *balance.second );
-                    [&acnt](const account_balance_object& balance) {
-                       acnt.balances.emplace_back(balance);
-                    });
 
       // Add the account's vesting balances
       auto vesting_range = _db.get_index_type<vesting_balance_index>().indices().get<by_account>().equal_range(account->id);
@@ -765,7 +764,7 @@ vector<account_id_type> database_api::get_account_references( account_id_type ac
 vector<account_id_type> database_api_impl::get_account_references( account_id_type account_id )const
 {
    const auto& idx = _db.get_index_type<account_index>();
-   const auto& aidx = dynamic_cast<const primary_index<account_index>&>(idx);
+   const auto& aidx = dynamic_cast<const base_primary_index&>(idx);
    const auto& refs = aidx.get_secondary_index<graphene::chain::account_member_index>();
    auto itr = refs.account_to_account_memberships.find(account_id);
    vector<account_id_type> result;
@@ -846,10 +845,10 @@ vector<asset> database_api_impl::get_account_balances(account_id_type acnt, cons
    if (assets.empty())
    {
       // if the caller passes in an empty list of assets, return balances for all assets the account owns
-      const account_balance_index& balance_index = _db.get_index_type<account_balance_index>();
+      const auto& balance_index = _db.get_index_type< primary_index< account_balance_index > >();
-      auto range = balance_index.indices().get<by_account_asset>().equal_range(boost::make_tuple(acnt));
+      const auto& balances = balance_index.get_secondary_index< balances_by_account_index >().get_account_balances( acnt );
-      for (const account_balance_object& balance : boost::make_iterator_range(range.first, range.second))
+      for( const auto balance : balances )
-         result.push_back(asset(balance.get_balance()));
+         result.push_back( balance.second->get_balance() );
    }
    else
    {
@@ -1005,7 +1004,7 @@ vector<optional<asset_object>> database_api_impl::lookup_asset_symbols(const vec
                   [this, &assets_by_symbol](const string& symbol_or_id) -> optional<asset_object> {
       if( !symbol_or_id.empty() && std::isdigit(symbol_or_id[0]) )
       {
-         auto ptr = _db.find(variant(symbol_or_id).as<asset_id_type>());
+         auto ptr = _db.find(variant(symbol_or_id, 1).as<asset_id_type>(1));
          return ptr == nullptr? optional<asset_object>() : *ptr;
       }
       auto itr = assets_by_symbol.find(symbol_or_id);
@@ -1014,6 +1013,112 @@ vector<optional<asset_object>> database_api_impl::lookup_asset_symbols(const vec
    return result;
 }
 
+uint64_t database_api::get_asset_count()const
+{
+   return my->get_asset_count();
+}
+
+uint64_t database_api_impl::get_asset_count()const
+{
+   return _db.get_index_type<asset_index>().indices().size();
+}
+////////////////////
+// Lottery Assets //
+////////////////////
+
+
+vector<asset_object> database_api::get_lotteries(  asset_id_type stop,
+                                                   unsigned limit,
+                                                   asset_id_type start )const
+{
+   return my->get_lotteries( stop, limit, start );
+}
+vector<asset_object> database_api_impl::get_lotteries( asset_id_type stop,
+                                                       unsigned limit,
+                                                       asset_id_type start )const
+{
+   vector<asset_object> result;
+   if( limit > 100 ) limit = 100;
+   const auto& assets = _db.get_index_type<asset_index>().indices().get<by_lottery>();
+
+   const auto range = assets.equal_range( boost::make_tuple( true ) );
+   for( const auto& a : boost::make_iterator_range( range.first, range.second ) )
+   {
+      if( start == asset_id_type() || (a.get_id().instance.value <= start.instance.value) )
+         result.push_back( a );
+      if( a.get_id().instance.value < stop.instance.value || result.size() >= limit )
+         break;
+   }
+
+   return result;
+}
+vector<asset_object> database_api::get_account_lotteries( account_id_type issuer, 
+                                                               asset_id_type stop,
+                                                               unsigned limit,
+                                                               asset_id_type start )const
+{
+   return my->get_account_lotteries( issuer, stop, limit, start );
+}
+
+vector<asset_object> database_api_impl::get_account_lotteries( account_id_type issuer, 
+                                                               asset_id_type stop,
+                                                               unsigned limit,
+                                                               asset_id_type start )const
+{
+   vector<asset_object> result;
+   if( limit > 100 ) limit = 100;
+   const auto& assets = _db.get_index_type<asset_index>().indices().get<by_lottery_owner>();
+
+   const auto range = assets.equal_range( boost::make_tuple( true, issuer.instance.value ) );
+   for( const auto& a : boost::make_iterator_range( range.first, range.second ) )
+   {
+      if( start == asset_id_type() || (a.get_id().instance.value <= start.instance.value) )
+         result.push_back( a );
+      if( a.get_id().instance.value < stop.instance.value || result.size() >= limit )
+         break;
+   }
+
+   return result;
+}
+
+asset database_api::get_lottery_balance( asset_id_type lottery_id )const
+{
+   return my->get_lottery_balance( lottery_id );
+}
+
+asset database_api_impl::get_lottery_balance( asset_id_type lottery_id )const
+{
+   auto lottery_asset = lottery_id( _db );
+   FC_ASSERT( lottery_asset.is_lottery() );
+   return _db.get_balance( lottery_id );
+}
+
+sweeps_vesting_balance_object database_api::get_sweeps_vesting_balance_object( account_id_type account )const
+{
+   return my->get_sweeps_vesting_balance_object( account );
+}
+
+sweeps_vesting_balance_object database_api_impl::get_sweeps_vesting_balance_object( account_id_type account )const
+{
+   const auto& vesting_idx = _db.get_index_type<sweeps_vesting_balance_index>().indices().get<by_owner>();
+   auto account_balance = vesting_idx.find(account);
+   FC_ASSERT( account_balance != vesting_idx.end(), "NO SWEEPS VESTING BALANCE" );
+   return *account_balance;
+}
+
+asset database_api::get_sweeps_vesting_balance_available_for_claim( account_id_type account )const
+{
+   return my->get_sweeps_vesting_balance_available_for_claim( account );
+}
+
+asset database_api_impl::get_sweeps_vesting_balance_available_for_claim( account_id_type account )const
+{
+   const auto& vesting_idx = _db.get_index_type<sweeps_vesting_balance_index>().indices().get<by_owner>();
+   auto account_balance = vesting_idx.find(account);
+   FC_ASSERT( account_balance != vesting_idx.end(), "NO SWEEPS VESTING BALANCE" );
+   return account_balance->available_for_claim();
+}
+
 //////////////////////////////////////////////////////////////////////
 // Peerplays                                                        //
 //////////////////////////////////////////////////////////////////////
@@ -1607,7 +1712,7 @@ vector<variant> database_api_impl::lookup_vote_ids( const vector<vote_id_type>&
          {
             auto itr = committee_idx.find( id );
             if( itr != committee_idx.end() )
-               result.emplace_back( variant( *itr ) );
+               result.emplace_back( variant( *itr, 1 ) );
             else
                result.emplace_back( variant() );
             break;
@@ -1616,7 +1721,7 @@ vector<variant> database_api_impl::lookup_vote_ids( const vector<vote_id_type>&
          {
             auto itr = witness_idx.find( id );
             if( itr != witness_idx.end() )
-               result.emplace_back( variant( *itr ) );
+               result.emplace_back( variant( *itr, 1 ) );
             else
                result.emplace_back( variant() );
             break;
@@ -1625,12 +1730,12 @@ vector<variant> database_api_impl::lookup_vote_ids( const vector<vote_id_type>&
          {
             auto itr = for_worker_idx.find( id );
             if( itr != for_worker_idx.end() ) {
-               result.emplace_back( variant( *itr ) );
+               result.emplace_back( variant( *itr, 1 ) );
             }
             else {
                auto itr = against_worker_idx.find( id );
                if( itr != against_worker_idx.end() ) {
-                  result.emplace_back( variant( *itr ) );
+                  result.emplace_back( variant( *itr, 1 ) );
                }
                else {
                   result.emplace_back( variant() );
@@ -1639,6 +1744,8 @@ vector<variant> database_api_impl::lookup_vote_ids( const vector<vote_id_type>&
             break;
          }
          case vote_id_type::VOTE_TYPE_COUNT: break; // supress unused enum value warnings
+         default:
+            FC_CAPTURE_AND_THROW( fc::out_of_range_exception, (id) );
       }
    }
    return result;
@@ -1747,8 +1854,8 @@ bool database_api::verify_authority( const signed_transaction& trx )const
 bool database_api_impl::verify_authority( const signed_transaction& trx )const
 {
    trx.verify_authority( _db.get_chain_id(),
-                         [&]( account_id_type id ){ return &id(_db).active; },
+                         [this]( account_id_type id ){ return &id(_db).active; },
-                         [&]( account_id_type id ){ return &id(_db).owner; },
+                         [this]( account_id_type id ){ return &id(_db).owner; },
                           _db.get_global_properties().parameters.max_authority_depth );
    return true;
 }
@@ -1763,7 +1870,7 @@ bool database_api_impl::verify_account_authority( const string& name_or_id, cons
    FC_ASSERT( name_or_id.size() > 0);
    const account_object* account = nullptr;
    if (std::isdigit(name_or_id[0]))
-      account = _db.find(fc::variant(name_or_id).as<account_id_type>());
+      account = _db.find(fc::variant(name_or_id, 1).as<account_id_type>(1));
    else
    {
       const auto& idx = _db.get_index_type<account_index>().indices().get<by_name>();
@@ -1824,7 +1931,7 @@ struct get_required_fees_helper
       {
          asset fee = current_fee_schedule.set_fee( op, core_exchange_rate );
          fc::variant result;
-         fc::to_variant( fee, result );
+         fc::to_variant( fee, result, GRAPHENE_NET_MAX_NESTED_OBJECTS );
          return result;
       }
    }
@@ -1844,7 +1951,7 @@ struct get_required_fees_helper
       // two mutually recursive functions instead of a visitor
       result.first = current_fee_schedule.set_fee( proposal_create_op, core_exchange_rate );
       fc::variant vresult;
-      fc::to_variant( result, vresult );
+      fc::to_variant( result, vresult, GRAPHENE_NET_MAX_NESTED_OBJECTS );
       return vresult;
    }
 
@@ -2023,55 +2130,6 @@ vector<tournament_id_type> database_api_impl::get_registered_tournaments(account
    return tournament_ids;
 }
 
-//////////////////////////////////////////////////////////////////////
-//                                                                  //
-// GPOS methods                                                     //
-//                                                                  //
-//////////////////////////////////////////////////////////////////////
-
-graphene::app::gpos_info database_api::get_gpos_info(const account_id_type account) const
-{
-   return my->get_gpos_info(account);
-
-}
-graphene::app::gpos_info database_api_impl::get_gpos_info(const account_id_type account) const
-{
-   gpos_info result;
-   result.vesting_factor = _db.calculate_vesting_factor(account(_db));
-
-   const auto& dividend_data = asset_id_type()(_db).dividend_data(_db);
-   const account_object& dividend_distribution_account = dividend_data.dividend_distribution_account(_db);
-   result.award = _db.get_balance(dividend_distribution_account, asset_id_type()(_db));
-
-   share_type total_amount;
-   auto balance_type = vesting_balance_type::gpos;
-#ifdef USE_VESTING_OBJECT_BY_ASSET_BALANCE_INDEX
-   // get only once a collection of accounts that hold nonzero vesting balances of the dividend asset
-   auto vesting_balances_begin =
-      vesting_index.indices().get<by_asset_balance>().lower_bound(boost::make_tuple(asset_id_type(), balance_type));
-   auto vesting_balances_end =
-      vesting_index.indices().get<by_asset_balance>().upper_bound(boost::make_tuple(asset_id_type(), balance_type, share_type()));
-
-   for (const vesting_balance_object& vesting_balance_obj : boost::make_iterator_range(vesting_balances_begin, vesting_balances_end))
-   {
-        total_amount += vesting_balance_obj.balance.amount;
-   }
-#else
-   const vesting_balance_index& vesting_index = _db.get_index_type<vesting_balance_index>();
-   const auto& vesting_balances = vesting_index.indices().get<by_id>();
-   for (const vesting_balance_object& vesting_balance_obj : vesting_balances)
-   {
-      if (vesting_balance_obj.balance.asset_id == asset_id_type() && vesting_balance_obj.balance_type == balance_type)
-      {
-         total_amount += vesting_balance_obj.balance.amount;
-      }
-   }
-#endif
-
-   result.total_amount = total_amount;
-   return result;
-}
-
 //////////////////////////////////////////////////////////////////////
 //                                                                  //
 // Private methods                                                  //
@@ -2155,7 +2213,7 @@ void database_api_impl::handle_object_changed(bool force_notify, bool full_objec
             }
             else
             {
-               updates.emplace_back( id );
+               updates.emplace_back( fc::variant( id, 1 ) );
             }
          }
       }
@@ -2199,7 +2257,7 @@ void database_api_impl::on_applied_block()
       auto capture_this = shared_from_this();
       block_id_type block_id = _db.head_block_id();
       fc::async([this,capture_this,block_id](){
-         _block_applied_callback(fc::variant(block_id));
+         _block_applied_callback(fc::variant(block_id, 1));
       });
    }
 
@@ -2240,7 +2298,7 @@ void database_api_impl::on_applied_block()
       {
          auto itr = _market_subscriptions.find(item.first);
          if(itr != _market_subscriptions.end())
-            itr->second(fc::variant(item.second));
+            itr->second(fc::variant(item.second, GRAPHENE_NET_MAX_NESTED_OBJECTS));
       }
    });
 }

libraries/app/impacted.cpp

@@ -282,6 +282,22 @@ struct get_impacted_account_visitor
       _impacted.insert( op.affiliate );
    }
    void operator()( const affiliate_referral_payout_operation& op ) { }
+   void operator()( const lottery_asset_create_operation& op) { }
+   void operator()( const ticket_purchase_operation& op )
+   {
+      _impacted.insert( op.buyer );
+   }
+   void operator()( const lottery_reward_operation& op ) {
+      _impacted.insert( op.winner );
+   }
+   void operator()( const lottery_end_operation& op ) {
+      for( auto participant : op.participants ) {
+         _impacted.insert(participant.first); 
+      }
+   }
+   void operator()( const sweeps_vesting_claim_operation& op ) { 
+      _impacted.insert( op.account );
+   }
 };
 
 void operation_get_impacted_accounts( const operation& op, flat_set<account_id_type>& result )

libraries/app/include/graphene/app/database_api.hpp

@@ -114,12 +114,6 @@ struct market_trade
    double                     value;
 };
 
-struct gpos_info {
-   double vesting_factor;
-   asset award;
-   share_type total_amount;
-};
-
 /**
  * @brief The database_api class implements the RPC API for the chain database.
  *
@@ -349,6 +343,34 @@ class database_api
        */
       vector<optional<asset_object>> lookup_asset_symbols(const vector<string>& symbols_or_ids)const;
 
+      /**
+       * @brief Get assets count
+       * @return The assets count
+       */
+      uint64_t get_asset_count()const;
+   
+      ////////////////////
+      // Lottery Assets //
+      ////////////////////
+      /**
+       * @brief Get a list of lottery assets
+       * @return The lottery assets between start and stop ids
+       */
+      vector<asset_object> get_lotteries( asset_id_type stop  = asset_id_type(),
+                                          unsigned limit = 100,
+                                          asset_id_type start = asset_id_type() )const;
+      vector<asset_object> get_account_lotteries( account_id_type issuer, 
+                                                  asset_id_type stop,
+                                                  unsigned limit,
+                                                  asset_id_type start )const;
+      sweeps_vesting_balance_object get_sweeps_vesting_balance_object( account_id_type account )const;
+      asset get_sweeps_vesting_balance_available_for_claim( account_id_type account )const;
+      /**
+       * @brief Get balance of lottery assets
+       */
+      asset get_lottery_balance( asset_id_type lottery_id ) const;
+   
+   
       /////////////////////
       // Peerplays       //
       /////////////////////
@@ -651,17 +673,7 @@ class database_api
        */
       vector<tournament_id_type> get_registered_tournaments(account_id_type account_filter, uint32_t limit) const;
 
-      //////////
+   private:
-      // GPOS //
-      //////////
-      /**
-       * @return account and network GPOS information
-       */
-      gpos_info get_gpos_info(const account_id_type account) const;
-
-
-
-private:
       std::shared_ptr< database_api_impl > my;
 };
 
@@ -672,8 +684,6 @@ FC_REFLECT( graphene::app::order_book, (base)(quote)(bids)(asks) );
 FC_REFLECT( graphene::app::market_ticker, (base)(quote)(latest)(lowest_ask)(highest_bid)(percent_change)(base_volume)(quote_volume) );
 FC_REFLECT( graphene::app::market_volume, (base)(quote)(base_volume)(quote_volume) );
 FC_REFLECT( graphene::app::market_trade, (date)(price)(amount)(value) );
-FC_REFLECT( graphene::app::gpos_info, (vesting_factor)(award)(total_amount) );
-
 
 FC_API(graphene::app::database_api,
    // Objects
@@ -723,6 +733,7 @@ FC_API(graphene::app::database_api,
    (get_assets)
    (list_assets)
    (lookup_asset_symbols)
+   (get_asset_count)
 
    // Peerplays
    (list_sports)
@@ -734,6 +745,13 @@ FC_API(graphene::app::database_api,
    (get_unmatched_bets_for_bettor)
    (get_all_unmatched_bets_for_bettor)
 
+   // Sweeps
+   (get_lotteries)
+   (get_account_lotteries)
+   (get_lottery_balance)
+   (get_sweeps_vesting_balance_object)
+   (get_sweeps_vesting_balance_available_for_claim)
+   
    // Markets / feeds
    (get_order_book)
    (get_limit_orders)
@@ -783,7 +801,4 @@ FC_API(graphene::app::database_api,
    (get_tournaments_by_state)
    (get_tournaments )
    (get_registered_tournaments)
-
-   // gpos
-   (get_gpos_info)
 )

libraries/app/include/graphene/app/plugin.hpp

@@ -121,16 +121,24 @@ class plugin : public abstract_plugin
 /// @group Some useful tools for boost::program_options arguments using vectors of JSON strings
 /// @{
 template<typename T>
-T dejsonify(const string& s)
+T dejsonify(const string& s, uint32_t max_depth)
 {
-   return fc::json::from_string(s).as<T>();
+   return fc::json::from_string(s).as<T>(max_depth);
+}
+
+namespace impl {
+   template<typename T>
+   T dejsonify( const string& s )
+   {
+      return graphene::app::dejsonify<T>( s, GRAPHENE_MAX_NESTED_OBJECTS );
+   }
 }
 
 #define DEFAULT_VALUE_VECTOR(value) default_value({fc::json::to_string(value)}, fc::json::to_string(value))
 #define LOAD_VALUE_SET(options, name, container, type) \
 if( options.count(name) ) { \
       const std::vector<std::string>& ops = options[name].as<std::vector<std::string>>(); \
-      std::transform(ops.begin(), ops.end(), std::inserter(container, container.end()), &graphene::app::dejsonify<type>); \
+      std::transform(ops.begin(), ops.end(), std::inserter(container, container.end()), &graphene::app::impl::dejsonify<type>); \
 }
 /// @}
 

libraries/chain/CMakeLists.txt

@@ -49,6 +49,7 @@ add_library( graphene_chain
              protocol/proposal.cpp
              protocol/withdraw_permission.cpp
              protocol/asset_ops.cpp
+             protocol/lottery_ops.cpp
              protocol/memo.cpp
              protocol/worker.cpp
              protocol/custom.cpp
@@ -72,6 +73,7 @@ add_library( graphene_chain
              witness_evaluator.cpp
              committee_member_evaluator.cpp
              asset_evaluator.cpp
+             lottery_evaluator.cpp
              transfer_evaluator.cpp
              proposal_evaluator.cpp
              market_evaluator.cpp

libraries/chain/account_object.cpp

@@ -119,9 +119,9 @@ set<account_id_type> account_member_index::get_account_members(const account_obj
       result.insert(auth.first);
    return result;
 }
-set<public_key_type> account_member_index::get_key_members(const account_object& a)const
+set<public_key_type, account_member_index::key_compare> account_member_index::get_key_members(const account_object& a)const
 {
-   set<public_key_type> result;
+   set<public_key_type, key_compare> result;
    for( auto auth : a.owner.key_auths )
       result.insert(auth.first);
    for( auto auth : a.active.key_auths )
@@ -213,7 +213,7 @@ void account_member_index::object_modified(const object& after)
 
 
     {
-       set<public_key_type> after_key_members = get_key_members(a);
+       set<public_key_type, key_compare> after_key_members = get_key_members(a);
 
        vector<public_key_type> removed; removed.reserve(before_key_members.size());
        std::set_difference(before_key_members.begin(), before_key_members.end(),
@@ -267,4 +267,54 @@ void account_referrer_index::object_modified( const object& after  )
 {
 }
 
+const uint8_t  balances_by_account_index::bits = 20;
+const uint64_t balances_by_account_index::mask = (1ULL << balances_by_account_index::bits) - 1;
+
+void balances_by_account_index::object_inserted( const object& obj )
+{
+   const auto& abo = dynamic_cast< const account_balance_object& >( obj );
+   while( balances.size() < (abo.owner.instance.value >> bits) + 1 )
+   {
+      balances.reserve( (abo.owner.instance.value >> bits) + 1 );
+      balances.resize( balances.size() + 1 );
+      balances.back().resize( 1ULL << bits );
+   }
+   balances[abo.owner.instance.value >> bits][abo.owner.instance.value & mask][abo.asset_type] = &abo;
+}
+
+void balances_by_account_index::object_removed( const object& obj )
+{
+   const auto& abo = dynamic_cast< const account_balance_object& >( obj );
+   if( balances.size() < (abo.owner.instance.value >> bits) + 1 ) return;
+   balances[abo.owner.instance.value >> bits][abo.owner.instance.value & mask].erase( abo.asset_type );
+}
+
+void balances_by_account_index::about_to_modify( const object& before )
+{
+   ids_being_modified.emplace( before.id );
+}
+
+void balances_by_account_index::object_modified( const object& after  )
+{
+   FC_ASSERT( ids_being_modified.top() == after.id, "Modification of ID is not supported!");
+   ids_being_modified.pop();
+}
+
+const map< asset_id_type, const account_balance_object* >& balances_by_account_index::get_account_balances( const account_id_type& acct )const
+{
+   static const map< asset_id_type, const account_balance_object* > _empty;
+
+   if( balances.size() < (acct.instance.value >> bits) + 1 ) return _empty;
+   return balances[acct.instance.value >> bits][acct.instance.value & mask];
+}
+
+const account_balance_object* balances_by_account_index::get_account_balance( const account_id_type& acct, const asset_id_type& asset )const
+{
+   if( balances.size() < (acct.instance.value >> bits) + 1 ) return nullptr;
+   const auto& mine = balances[acct.instance.value >> bits][acct.instance.value & mask];
+   const auto itr = mine.find( asset );
+   if( mine.end() == itr ) return nullptr;
+   return itr->second;
+}
+
 } } // graphene::chain

libraries/chain/asset_evaluator.cpp

@@ -24,6 +24,7 @@
 #include <graphene/chain/asset_evaluator.hpp>
 #include <graphene/chain/asset_object.hpp>
 #include <graphene/chain/account_object.hpp>
+#include <graphene/chain/balance_object.hpp>
 #include <graphene/chain/market_object.hpp>
 #include <graphene/chain/database.hpp>
 #include <graphene/chain/exceptions.hpp>
@@ -75,6 +76,7 @@ void_result asset_create_evaluator::do_evaluate( const asset_create_operation& o
    {
       auto dotpos = op.symbol.rfind( '.' );
       if( dotpos != std::string::npos )
+      
       {
          auto prefix = op.symbol.substr( 0, dotpos );
          auto asset_symbol_itr = asset_indx.find( prefix );
@@ -119,6 +121,7 @@ void_result asset_create_evaluator::do_evaluate( const asset_create_operation& o
    return void_result();
 } FC_CAPTURE_AND_RETHROW( (op) ) }
 
+// copied from bitshares. (https://github.com/bitshares/bitshares-core/issues/429)
 void asset_create_evaluator::pay_fee()
 {
    fee_is_odd = core_fee_paid.value & 1;
@@ -127,6 +130,154 @@ void asset_create_evaluator::pay_fee()
 }
 
 object_id_type asset_create_evaluator::do_apply( const asset_create_operation& op )
+{ try {
+   // includes changes from bitshares. (https://github.com/bitshares/bitshares-core/issues/429)
+   bool hf_429 = fee_is_odd && db().head_block_time() > HARDFORK_CORE_429_TIME;
+
+   const asset_dynamic_data_object& dyn_asset =
+      db().create<asset_dynamic_data_object>( [&]( asset_dynamic_data_object& a ) {
+         a.current_supply = 0;
+         a.fee_pool = core_fee_paid - (hf_429 ? 1 : 0);
+      });
+      if( fee_is_odd && !hf_429 )
+      {
+         const auto& core_dd = db().get<asset_object>( asset_id_type() ).dynamic_data( db() );
+         db().modify( core_dd, [=]( asset_dynamic_data_object& dd ) {
+            dd.current_supply++;
+         });
+      }
+
+   asset_bitasset_data_id_type bit_asset_id;
+   if( op.bitasset_opts.valid() )
+      bit_asset_id = db().create<asset_bitasset_data_object>( [&]( asset_bitasset_data_object& a ) {
+            a.options = *op.bitasset_opts;
+            a.is_prediction_market = op.is_prediction_market;
+         }).id;
+
+   auto next_asset_id = db().get_index_type<asset_index>().get_next_id();
+
+   const asset_object& new_asset =
+     db().create<asset_object>( [&]( asset_object& a ) {
+         a.issuer = op.issuer;
+         a.symbol = op.symbol;
+         a.precision = op.precision;
+         a.options = op.common_options;
+
+         if( a.options.core_exchange_rate.base.asset_id.instance.value == 0 )
+            a.options.core_exchange_rate.quote.asset_id = next_asset_id;
+         else
+            a.options.core_exchange_rate.base.asset_id = next_asset_id;
+
+         a.dynamic_asset_data_id = dyn_asset.id;
+         
+         if( op.bitasset_opts.valid() )
+            a.bitasset_data_id = bit_asset_id;
+      });
+   assert( new_asset.id == next_asset_id );
+
+   return new_asset.id;
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+void_result lottery_asset_create_evaluator::do_evaluate( const lottery_asset_create_operation& op )
+{ try {
+
+   database& d = db();
+
+   const auto& chain_parameters = d.get_global_properties().parameters;
+   FC_ASSERT( op.common_options.whitelist_authorities.size() <= chain_parameters.maximum_asset_whitelist_authorities );
+   FC_ASSERT( op.common_options.blacklist_authorities.size() <= chain_parameters.maximum_asset_whitelist_authorities );
+
+   // Check that all authorities do exist
+   for( auto id : op.common_options.whitelist_authorities )
+      d.get_object(id);
+   for( auto id : op.common_options.blacklist_authorities )
+      d.get_object(id);
+
+   auto& asset_indx = d.get_index_type<asset_index>().indices().get<by_symbol>();
+   auto asset_symbol_itr = asset_indx.find( op.symbol );
+   FC_ASSERT( asset_symbol_itr == asset_indx.end() );
+
+   if( d.head_block_time() > HARDFORK_385_TIME )
+   {
+
+   if( d.head_block_time() <= HARDFORK_409_TIME )
+   {
+      auto dotpos = op.symbol.find( '.' );
+      if( dotpos != std::string::npos )
+      {
+         auto prefix = op.symbol.substr( 0, dotpos );
+         auto asset_symbol_itr = asset_indx.find( op.symbol );
+         FC_ASSERT( asset_symbol_itr != asset_indx.end(), "Asset ${s} may only be created by issuer of ${p}, but ${p} has not been registered",
+                    ("s",op.symbol)("p",prefix) );
+         FC_ASSERT( asset_symbol_itr->issuer == op.issuer, "Asset ${s} may only be created by issuer of ${p}, ${i}",
+                    ("s",op.symbol)("p",prefix)("i", op.issuer(d).name) );
+      }
+   }
+   else
+   {
+      auto dotpos = op.symbol.rfind( '.' );
+      if( dotpos != std::string::npos )
+      
+      {
+         auto prefix = op.symbol.substr( 0, dotpos );
+         auto asset_symbol_itr = asset_indx.find( prefix );
+         FC_ASSERT( asset_symbol_itr != asset_indx.end(), "Asset ${s} may only be created by issuer of ${p}, but ${p} has not been registered",
+                    ("s",op.symbol)("p",prefix) );
+         FC_ASSERT( asset_symbol_itr->issuer == op.issuer, "Asset ${s} may only be created by issuer of ${p}, ${i}",
+                    ("s",op.symbol)("p",prefix)("i", op.issuer(d).name) );
+      }
+   }
+
+   }
+   else
+   {
+      auto dotpos = op.symbol.find( '.' );
+      if( dotpos != std::string::npos )
+          wlog( "Asset ${s} has a name which requires hardfork 385", ("s",op.symbol) );
+   }
+
+   // core_fee_paid -= core_fee_paid.value/2;
+
+   if( op.bitasset_opts )
+   {
+      const asset_object& backing = op.bitasset_opts->short_backing_asset(d);
+      if( backing.is_market_issued() )
+      {
+         const asset_bitasset_data_object& backing_bitasset_data = backing.bitasset_data(d);
+         const asset_object& backing_backing = backing_bitasset_data.options.short_backing_asset(d);
+         FC_ASSERT( !backing_backing.is_market_issued(),
+                    "May not create a bitasset backed by a bitasset backed by a bitasset." );
+         FC_ASSERT( op.issuer != GRAPHENE_COMMITTEE_ACCOUNT || backing_backing.get_id() == asset_id_type(),
+                    "May not create a blockchain-controlled market asset which is not backed by CORE.");
+      } else
+         FC_ASSERT( op.issuer != GRAPHENE_COMMITTEE_ACCOUNT || backing.get_id() == asset_id_type(),
+                    "May not create a blockchain-controlled market asset which is not backed by CORE.");
+      FC_ASSERT( op.bitasset_opts->feed_lifetime_sec > chain_parameters.block_interval &&
+                 op.bitasset_opts->force_settlement_delay_sec > chain_parameters.block_interval );
+   }
+   if( op.is_prediction_market )
+   {
+      FC_ASSERT( op.bitasset_opts );
+      FC_ASSERT( op.precision == op.bitasset_opts->short_backing_asset(d).precision );
+   }
+
+   FC_ASSERT( op.common_options.max_supply >= 5 );
+   auto lottery_options = op.extensions;
+   lottery_options.validate();
+   FC_ASSERT( lottery_options.end_date > d.head_block_time() || lottery_options.end_date == time_point_sec() );
+
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+// copied from bitshares. (https://github.com/bitshares/bitshares-core/issues/429)
+void lottery_asset_create_evaluator::pay_fee()
+{
+   fee_is_odd = core_fee_paid.value & 1;
+   core_fee_paid -= core_fee_paid.value/2;
+   generic_evaluator::pay_fee();
+}
+
+object_id_type lottery_asset_create_evaluator::do_apply( const lottery_asset_create_operation& op )
 { try {
    const asset_dynamic_data_object& dyn_asset =
       db().create<asset_dynamic_data_object>( [&]( asset_dynamic_data_object& a ) {
@@ -149,6 +300,13 @@ object_id_type asset_create_evaluator::do_apply( const asset_create_operation& o
          a.symbol = op.symbol;
          a.precision = op.precision;
          a.options = op.common_options;
+         a.precision = 0;
+         a.lottery_options = op.extensions;
+         //a.lottery_options->balance = asset( 0, a.lottery_options->ticket_price.asset_id );
+         a.lottery_options->owner = a.id;
+         db().create<lottery_balance_object>([&](lottery_balance_object& lbo) {
+            lbo.lottery_id = a.id;
+         });
          if( a.options.core_exchange_rate.base.asset_id.instance.value == 0 )
             a.options.core_exchange_rate.quote.asset_id = next_asset_id;
          else
@@ -169,6 +327,7 @@ void_result asset_issue_evaluator::do_evaluate( const asset_issue_operation& o )
    const asset_object& a = o.asset_to_issue.asset_id(d);
    FC_ASSERT( o.issuer == a.issuer );
    FC_ASSERT( !a.is_market_issued(), "Cannot manually issue a market-issued asset." );
+   FC_ASSERT( !a.is_lottery(), "Cannot manually issue a lottery asset." );
 
    to_account = &o.issue_to_account(d);
    FC_ASSERT( is_authorized_asset( d, *to_account, a ) );

libraries/chain/asset_object.cpp

@@ -43,7 +43,7 @@ share_type asset_bitasset_data_object::max_force_settlement_volume(share_type cu
    return volume.to_uint64();
 }
 
-void graphene::chain::asset_bitasset_data_object::update_median_feeds(time_point_sec current_time)
+void asset_bitasset_data_object::update_median_feeds(time_point_sec current_time)
 {
    current_feed_publication_time = current_time;
    vector<std::reference_wrapper<const price_feed>> current_feeds;
@@ -89,6 +89,12 @@ void graphene::chain::asset_bitasset_data_object::update_median_feeds(time_point
 }
 
 
+time_point_sec asset_object::get_lottery_expiration() const 
+{
+   if( lottery_options )
+      return lottery_options->end_date;
+   return time_point_sec();
+}
 
 asset asset_object::amount_from_string(string amount_string) const
 { try {
@@ -158,3 +164,130 @@ string asset_object::amount_to_string(share_type amount) const
       result += "." + fc::to_string(scaled_precision.value + decimals).erase(0,1);
    return result;
 }
+
+
+vector<account_id_type> asset_object::get_holders( database& db ) const
+{
+   auto& asset_bal_idx = db.get_index_type< account_balance_index >().indices().get< by_asset_balance >();
+   
+   uint64_t max_supply = get_id()(db).options.max_supply.value;
+   
+   vector<account_id_type> holders; // repeating if balance > 1
+   holders.reserve(max_supply);
+   const auto range = asset_bal_idx.equal_range( boost::make_tuple( get_id() ) );
+   for( const account_balance_object& bal : boost::make_iterator_range( range.first, range.second ) )
+      for( uint64_t balance = bal.balance.value; balance > 0; --balance)
+         holders.push_back( bal.owner );
+   return holders;
+}
+
+void asset_object::distribute_benefactors_part( database& db )
+{
+   transaction_evaluation_state eval( &db );
+   uint64_t jackpot = get_id()( db ).dynamic_data( db ).current_supply.value * lottery_options->ticket_price.amount.value;
+   
+   for( auto benefactor : lottery_options->benefactors ) {
+      lottery_reward_operation reward_op;
+      reward_op.lottery = get_id();
+      reward_op.winner = benefactor.id;
+      reward_op.is_benefactor_reward = true;
+      reward_op.win_percentage = benefactor.share;
+      reward_op.amount = asset( jackpot * benefactor.share / GRAPHENE_100_PERCENT, db.get_balance(id).asset_id );
+      db.apply_operation(eval, reward_op);
+   }
+}
+
+map< account_id_type, vector< uint16_t > > asset_object::distribute_winners_part( database& db )
+{
+   transaction_evaluation_state eval( &db );
+      
+   auto holders = get_holders( db );
+   FC_ASSERT( dynamic_data( db ).current_supply == holders.size() );
+   map<account_id_type, vector<uint16_t> > structurized_participants;
+   for( account_id_type holder : holders )
+   {
+      if( !structurized_participants.count( holder ) )
+         structurized_participants.emplace( holder, vector< uint16_t >() );
+   }
+   uint64_t jackpot = get_id()( db ).dynamic_data( db ).current_supply.value * lottery_options->ticket_price.amount.value;
+   auto winner_numbers = db.get_winner_numbers( get_id(), holders.size(), lottery_options->winning_tickets.size() );
+   
+   auto& tickets( lottery_options->winning_tickets );
+   
+   if( holders.size() < tickets.size() ) {
+      uint16_t percents_to_distribute = 0;
+      for( auto i = tickets.begin() + holders.size(); i != tickets.end(); ) {
+         percents_to_distribute += *i;
+         i = tickets.erase(i);
+      }
+      for( auto t = tickets.begin(); t != tickets.begin() + holders.size(); ++t )
+         *t += percents_to_distribute / holders.size();
+   }
+   auto sweeps_distribution_percentage = db.get_global_properties().parameters.sweeps_distribution_percentage();
+   for( int c = 0; c < winner_numbers.size(); ++c ) {
+      auto winner_num = winner_numbers[c];
+      lottery_reward_operation reward_op;
+      reward_op.lottery = get_id();
+      reward_op.is_benefactor_reward = false;
+      reward_op.winner = holders[winner_num];
+      reward_op.win_percentage = tickets[c];
+      reward_op.amount = asset( jackpot * tickets[c] * ( 1. - sweeps_distribution_percentage / (double)GRAPHENE_100_PERCENT ) / GRAPHENE_100_PERCENT , db.get_balance(id).asset_id );
+      db.apply_operation(eval, reward_op);
+      
+      structurized_participants[ holders[ winner_num ] ].push_back( tickets[c] );
+   }
+   return structurized_participants;
+}
+
+void asset_object::distribute_sweeps_holders_part( database& db )
+{
+   transaction_evaluation_state eval( &db );
+   
+   auto& asset_bal_idx = db.get_index_type< account_balance_index >().indices().get< by_asset_balance >();
+   
+   auto sweeps_params = db.get_global_properties().parameters;
+   uint64_t distribution_asset_supply = sweeps_params.sweeps_distribution_asset()( db ).dynamic_data( db ).current_supply.value;
+   const auto range = asset_bal_idx.equal_range( boost::make_tuple( sweeps_params.sweeps_distribution_asset() ) );
+   
+   uint64_t holders_sum = 0;
+   for( const account_balance_object& holder_balance : boost::make_iterator_range( range.first, range.second ) )
+   {
+      int64_t holder_part = db.get_balance(id).amount.value / (double)distribution_asset_supply * holder_balance.balance.value * SWEEPS_VESTING_BALANCE_MULTIPLIER;
+      db.adjust_sweeps_vesting_balance( holder_balance.owner, holder_part );
+      holders_sum += holder_part;
+   }
+   uint64_t balance_rest = db.get_balance( get_id() ).amount.value * SWEEPS_VESTING_BALANCE_MULTIPLIER - holders_sum;
+   db.adjust_sweeps_vesting_balance( sweeps_params.sweeps_vesting_accumulator_account(), balance_rest );
+   db.adjust_balance( get_id(), -db.get_balance( get_id() ) );
+}
+
+void asset_object::end_lottery( database& db )
+{
+   transaction_evaluation_state eval(&db);
+   
+   FC_ASSERT( is_lottery() );
+   FC_ASSERT( lottery_options->is_active && ( lottery_options->end_date <= db.head_block_time() || lottery_options->ending_on_soldout ) );
+
+   auto participants = distribute_winners_part( db );
+   if( participants.size() > 0) {
+      distribute_benefactors_part( db );
+      distribute_sweeps_holders_part( db );
+   }
+   
+   lottery_end_operation end_op;
+   end_op.lottery = id;
+   end_op.participants = participants;
+   db.apply_operation(eval, end_op);
+}
+
+void lottery_balance_object::adjust_balance( const asset& delta )
+{
+   FC_ASSERT( delta.asset_id == balance.asset_id );
+   balance += delta;
+}
+
+void sweeps_vesting_balance_object::adjust_balance( const asset& delta )
+{
+   FC_ASSERT( delta.asset_id == asset_id );
+   balance += delta.amount.value;
+}

libraries/chain/betting_market_evaluator.cpp

@@ -340,7 +340,7 @@ object_id_type bet_place_evaluator::do_apply(const bet_place_operation& op)
               ("balance", d.get_balance(*fee_paying_account, *_asset))("amount_to_bet", op.amount_to_bet.amount)  );
 
    // pay for it
-   d.adjust_balance(fee_paying_account->id, -op.amount_to_bet);
+   d.adjust_balance(fee_paying_account->id.as<account_id_type>(), -op.amount_to_bet);
 
    return new_bet_id;
 } FC_CAPTURE_AND_RETHROW( (op) ) }

libraries/chain/betting_market_group_object.cpp

@@ -26,7 +26,7 @@
 #include <graphene/chain/betting_market_object.hpp>
 #include <graphene/chain/event_object.hpp>
 #include <graphene/chain/database.hpp>
-#include <boost/integer/common_factor_rt.hpp>
+#include <boost/math/common_factor_rt.hpp>
 
 #include <boost/msm/back/state_machine.hpp>
 #include <boost/msm/front/state_machine_def.hpp>
@@ -543,35 +543,35 @@ void betting_market_group_object::dispatch_new_status(database& db, betting_mark
 
 namespace fc { 
    // Manually reflect betting_market_group_object to variant to properly reflect "state"
-   void to_variant(const graphene::chain::betting_market_group_object& betting_market_group_obj, fc::variant& v)
+   void to_variant(const graphene::chain::betting_market_group_object& betting_market_group_obj, fc::variant& v, uint32_t max_depth)
    {
       fc::mutable_variant_object o;
-      o("id", betting_market_group_obj.id)
+      o("id", fc::variant(betting_market_group_obj.id, max_depth))
-       ("description", betting_market_group_obj.description)
+       ("description", fc::variant(betting_market_group_obj.description, max_depth))
-       ("event_id", betting_market_group_obj.event_id)
+       ("event_id", fc::variant(betting_market_group_obj.event_id, max_depth))
-       ("rules_id", betting_market_group_obj.rules_id)
+       ("rules_id", fc::variant(betting_market_group_obj.rules_id, max_depth))
-       ("asset_id", betting_market_group_obj.asset_id)
+       ("asset_id", fc::variant(betting_market_group_obj.asset_id, max_depth))
-       ("total_matched_bets_amount", betting_market_group_obj.total_matched_bets_amount)
+       ("total_matched_bets_amount", fc::variant(betting_market_group_obj.total_matched_bets_amount, max_depth))
-       ("never_in_play", betting_market_group_obj.never_in_play)
+       ("never_in_play", fc::variant(betting_market_group_obj.never_in_play, max_depth))
-       ("delay_before_settling", betting_market_group_obj.delay_before_settling)
+       ("delay_before_settling", fc::variant(betting_market_group_obj.delay_before_settling, max_depth))
-       ("settling_time", betting_market_group_obj.settling_time)
+       ("settling_time", fc::variant(betting_market_group_obj.settling_time, max_depth))
-       ("status", betting_market_group_obj.get_status());
+       ("status", fc::variant(betting_market_group_obj.get_status(), max_depth));
 
       v = o;
    }
 
    // Manually reflect betting_market_group_object to variant to properly reflect "state"
-   void from_variant(const fc::variant& v, graphene::chain::betting_market_group_object& betting_market_group_obj)
+   void from_variant(const fc::variant& v, graphene::chain::betting_market_group_object& betting_market_group_obj, uint32_t max_depth)
    {
-      betting_market_group_obj.id = v["id"].as<graphene::chain::betting_market_group_id_type>();
+      betting_market_group_obj.id = v["id"].as<graphene::chain::betting_market_group_id_type>( max_depth );
-      betting_market_group_obj.description = v["description"].as<graphene::chain::internationalized_string_type>();
+      betting_market_group_obj.description = v["description"].as<graphene::chain::internationalized_string_type>( max_depth );
-      betting_market_group_obj.event_id = v["event_id"].as<graphene::chain::event_id_type>();
+      betting_market_group_obj.event_id = v["event_id"].as<graphene::chain::event_id_type>( max_depth );
-      betting_market_group_obj.asset_id = v["asset_id"].as<graphene::chain::asset_id_type>();
+      betting_market_group_obj.asset_id = v["asset_id"].as<graphene::chain::asset_id_type>( max_depth );
-      betting_market_group_obj.total_matched_bets_amount = v["total_matched_bets_amount"].as<graphene::chain::share_type>();
+      betting_market_group_obj.total_matched_bets_amount = v["total_matched_bets_amount"].as<graphene::chain::share_type>( max_depth );
-      betting_market_group_obj.never_in_play = v["never_in_play"].as<bool>();
+      betting_market_group_obj.never_in_play = v["never_in_play"].as<bool>( max_depth );
-      betting_market_group_obj.delay_before_settling = v["delay_before_settling"].as<uint32_t>();
+      betting_market_group_obj.delay_before_settling = v["delay_before_settling"].as<uint32_t>( max_depth );
-      betting_market_group_obj.settling_time = v["settling_time"].as<fc::optional<fc::time_point_sec>>();
+      betting_market_group_obj.settling_time = v["settling_time"].as<fc::optional<fc::time_point_sec>>( max_depth );
-      graphene::chain::betting_market_group_status status = v["status"].as<graphene::chain::betting_market_group_status>();
+      graphene::chain::betting_market_group_status status = v["status"].as<graphene::chain::betting_market_group_status>( max_depth );
       const_cast<int*>(betting_market_group_obj.my->state_machine.current_state())[0] = (int)status;
    }
 } //end namespace fc

libraries/chain/betting_market_object.cpp

@@ -468,28 +468,28 @@ void betting_market_object::on_canceled_event(database& db)
 
 namespace fc { 
    // Manually reflect betting_market_object to variant to properly reflect "state"
-   void to_variant(const graphene::chain::betting_market_object& event_obj, fc::variant& v)
+   void to_variant(const graphene::chain::betting_market_object& event_obj, fc::variant& v, uint32_t max_depth)
    {
       fc::mutable_variant_object o;
-      o("id", event_obj.id)
+      o("id", fc::variant(event_obj.id, max_depth) )
-       ("group_id", event_obj.group_id)
+       ("group_id", fc::variant(event_obj.group_id, max_depth))
-       ("description", event_obj.description)
+       ("description", fc::variant(event_obj.description, max_depth))
-       ("payout_condition", event_obj.payout_condition)
+       ("payout_condition", fc::variant(event_obj.payout_condition, max_depth))
-       ("resolution", event_obj.resolution)
+       ("resolution", fc::variant(event_obj.resolution, max_depth))
-       ("status", event_obj.get_status());
+       ("status", fc::variant(event_obj.get_status(), max_depth));
 
       v = o;
    }
 
    // Manually reflect betting_market_object to variant to properly reflect "state"
-   void from_variant(const fc::variant& v, graphene::chain::betting_market_object& event_obj)
+   void from_variant(const fc::variant& v, graphene::chain::betting_market_object& event_obj, uint32_t max_depth)
    {
-      event_obj.id = v["id"].as<graphene::chain::betting_market_id_type>();
+      event_obj.id = v["id"].as<graphene::chain::betting_market_id_type>( max_depth );
-      event_obj.group_id = v["name"].as<graphene::chain::betting_market_group_id_type>();
+      event_obj.group_id = v["name"].as<graphene::chain::betting_market_group_id_type>( max_depth );
-      event_obj.description = v["description"].as<graphene::chain::internationalized_string_type>();
+      event_obj.description = v["description"].as<graphene::chain::internationalized_string_type>( max_depth );
-      event_obj.payout_condition = v["payout_condition"].as<graphene::chain::internationalized_string_type>();
+      event_obj.payout_condition = v["payout_condition"].as<graphene::chain::internationalized_string_type>( max_depth );
-      event_obj.resolution = v["resolution"].as<fc::optional<graphene::chain::betting_market_resolution_type>>();
+      event_obj.resolution = v["resolution"].as<fc::optional<graphene::chain::betting_market_resolution_type>>( max_depth );
-      graphene::chain::betting_market_status status = v["status"].as<graphene::chain::betting_market_status>();
+      graphene::chain::betting_market_status status = v["status"].as<graphene::chain::betting_market_status>( max_depth );
       const_cast<int*>(event_obj.my->state_machine.current_state())[0] = (int)status;
    }
 } //end namespace fc

libraries/chain/block_database.cpp

@@ -45,14 +45,15 @@ void block_database::open( const fc::path& dbdir )
    _block_num_to_pos.exceptions(std::ios_base::failbit | std::ios_base::badbit);
    _blocks.exceptions(std::ios_base::failbit | std::ios_base::badbit);
 
-   if( !fc::exists( dbdir/"index" ) )
+   _index_filename = dbdir / "index";
+   if( !fc::exists( _index_filename ) )
    {
-     _block_num_to_pos.open( (dbdir/"index").generic_string().c_str(), std::fstream::binary | std::fstream::in | std::fstream::out | std::fstream::trunc);
+     _block_num_to_pos.open( _index_filename.generic_string().c_str(), std::fstream::binary | std::fstream::in | std::fstream::out | std::fstream::trunc);
      _blocks.open( (dbdir/"blocks").generic_string().c_str(), std::fstream::binary | std::fstream::in | std::fstream::out | std::fstream::trunc);
    }
    else
    {
-     _block_num_to_pos.open( (dbdir/"index").generic_string().c_str(), std::fstream::binary | std::fstream::in | std::fstream::out );
+     _block_num_to_pos.open( _index_filename.generic_string().c_str(), std::fstream::binary | std::fstream::in | std::fstream::out );
      _blocks.open( (dbdir/"blocks").generic_string().c_str(), std::fstream::binary | std::fstream::in | std::fstream::out );
    }
 } FC_CAPTURE_AND_RETHROW( (dbdir) ) }
@@ -121,7 +122,7 @@ bool block_database::contains( const block_id_type& id )const
    index_entry e;
    auto index_pos = sizeof(e)*block_header::num_from_id(id);
    _block_num_to_pos.seekg( 0, _block_num_to_pos.end );
-   if ( _block_num_to_pos.tellg() <= index_pos )
+   if ( _block_num_to_pos.tellg() < index_pos + sizeof(e) )
       return false;
    _block_num_to_pos.seekg( index_pos );
    _block_num_to_pos.read( (char*)&e, sizeof(e) );
@@ -206,34 +207,38 @@ optional<signed_block> block_database::fetch_by_number( uint32_t block_num )cons
    return optional<signed_block>();
 }
 
-optional<signed_block> block_database::last()const
+optional<index_entry> block_database::last_index_entry()const {
-{
    try
    {
       index_entry e;
+
       _block_num_to_pos.seekg( 0, _block_num_to_pos.end );
+      std::streampos pos = _block_num_to_pos.tellg();
+      if( pos < sizeof(index_entry) )
+         return optional<index_entry>();
 
-      if( _block_num_to_pos.tellp() < sizeof(index_entry) )
+      pos -= pos % sizeof(index_entry);
-         return optional<signed_block>();
 
-      _block_num_to_pos.seekg( -sizeof(index_entry), _block_num_to_pos.end );
+      _blocks.seekg( 0, _block_num_to_pos.end );
-      _block_num_to_pos.read( (char*)&e, sizeof(e) );
+      const std::streampos blocks_size = _blocks.tellg();
-      uint64_t pos = _block_num_to_pos.tellg();
+      while( pos > 0 )
-      while( e.block_size == 0 && pos > 0 )
       {
          pos -= sizeof(index_entry);
          _block_num_to_pos.seekg( pos );
          _block_num_to_pos.read( (char*)&e, sizeof(e) );
-      }
+         if( _block_num_to_pos.gcount() == sizeof(e) && e.block_size > 0
-
+                && e.block_pos + e.block_size <= blocks_size )
-      if( e.block_size == 0 )
+            try
-         return optional<signed_block>();
+            {
-
                vector<char> data( e.block_size );
                _blocks.seekg( e.block_pos );
                _blocks.read( data.data(), e.block_size );
-      auto result = fc::raw::unpack<signed_block>(data);
+               if( _blocks.gcount() == e.block_size )
-      return result;
+               {
+                  const signed_block block = fc::raw::unpack<signed_block>(data);
+                  if( block.id() == e.block_id )
+                     return e;
+               }
             }
             catch (const fc::exception&)
             {
@@ -241,42 +246,30 @@ optional<signed_block> block_database::last()const
             catch (const std::exception&)
             {
             }
+         fc::resize_file( _index_filename, pos );
+      }
+   }
+   catch (const fc::exception&)
+   {
+   }
+   catch (const std::exception&)
+   {
+   }
+   return optional<index_entry>();
+}
+
+optional<signed_block> block_database::last()const
+{
+   optional<index_entry> entry = last_index_entry();
+   if( entry.valid() ) return fetch_by_number( block_header::num_from_id(entry->block_id) );
    return optional<signed_block>();
 }
 
 optional<block_id_type> block_database::last_id()const
 {
-   try
+   optional<index_entry> entry = last_index_entry();
-   {
+   if( entry.valid() ) return entry->block_id;
-      index_entry e;
-      _block_num_to_pos.seekg( 0, _block_num_to_pos.end );
-
-      if( _block_num_to_pos.tellp() < sizeof(index_entry) )
-         return optional<block_id_type>();
-
-      _block_num_to_pos.seekg( -sizeof(index_entry), _block_num_to_pos.end );
-      _block_num_to_pos.read( (char*)&e, sizeof(e) );
-      uint64_t pos = _block_num_to_pos.tellg();
-      while( e.block_size == 0 && pos > 0 )
-      {
-         pos -= sizeof(index_entry);
-         _block_num_to_pos.seekg( pos );
-         _block_num_to_pos.read( (char*)&e, sizeof(e) );
-      }
-
-      if( e.block_size == 0 )
-         return optional<block_id_type>();
-
-      return e.block_id;
-   }
-   catch (const fc::exception&)
-   {
-   }
-   catch (const std::exception&)
-   {
-   }
    return optional<block_id_type>();
 }
 
-
 } }

libraries/chain/db_balance.cpp

@@ -26,6 +26,7 @@
 
 #include <graphene/chain/account_object.hpp>
 #include <graphene/chain/asset_object.hpp>
+#include <graphene/chain/balance_object.hpp>
 #include <graphene/chain/vesting_balance_object.hpp>
 #include <graphene/chain/witness_object.hpp>
 
@@ -33,11 +34,11 @@ namespace graphene { namespace chain {
 
 asset database::get_balance(account_id_type owner, asset_id_type asset_id) const
 {
-   auto& index = get_index_type<account_balance_index>().indices().get<by_account_asset>();
+   auto& index = get_index_type< primary_index< account_balance_index > >().get_secondary_index<balances_by_account_index>();
-   auto itr = index.find(boost::make_tuple(owner, asset_id));
+   auto abo = index.get_account_balance( owner, asset_id );
-   if( itr == index.end() )
+   if( !abo )
       return asset(0, asset_id);
-   return itr->get_balance();
+   return abo->get_balance();
 }
 
 asset database::get_balance(const account_object& owner, const asset_object& asset_obj) const
@@ -45,6 +46,15 @@ asset database::get_balance(const account_object& owner, const asset_object& ass
    return get_balance(owner.get_id(), asset_obj.get_id());
 }
 
+asset database::get_balance(asset_id_type lottery_id)const
+{
+   auto& index = get_index_type<lottery_balance_index>().indices().get<by_owner>();
+   auto itr = index.find( lottery_id );
+   if( itr == index.end() )
+      return asset(0, asset_id_type( ));
+   return itr->get_balance();
+}
+
 string database::to_pretty_string( const asset& a )const
 {
    return a.asset_id(*this).amount_to_pretty_string(a.amount);
@@ -55,9 +65,9 @@ void database::adjust_balance(account_id_type account, asset delta )
    if( delta.amount == 0 )
       return;
 
-   auto& index = get_index_type<account_balance_index>().indices().get<by_account_asset>();
+   auto& index = get_index_type< primary_index< account_balance_index > >().get_secondary_index<balances_by_account_index>();
-   auto itr = index.find(boost::make_tuple(account, delta.asset_id));
+   auto abo = index.get_account_balance( account, delta.asset_id );
-   if(itr == index.end())
+   if( !abo )
    {
       FC_ASSERT( delta.amount > 0, "Insufficient Balance: ${a}'s balance of ${b} is less than required ${r}", 
                  ("a",account(*this).name)
@@ -70,14 +80,73 @@ void database::adjust_balance(account_id_type account, asset delta )
       });
    } else {
       if( delta.amount < 0 )
-         FC_ASSERT( itr->get_balance() >= -delta, "Insufficient Balance: ${a}'s balance of ${b} is less than required ${r}", ("a",account(*this).name)("b",to_pretty_string(itr->get_balance()))("r",to_pretty_string(-delta)));
+         FC_ASSERT( abo->get_balance() >= -delta, "Insufficient Balance: ${a}'s balance of ${b} is less than required ${r}",
-      modify(*itr, [delta](account_balance_object& b) {
+                    ("a",account(*this).name)("b",to_pretty_string(abo->get_balance()))("r",to_pretty_string(-delta)));
+      modify(*abo, [delta](account_balance_object& b) {
          b.adjust_balance(delta);
       });
    }
 
 } FC_CAPTURE_AND_RETHROW( (account)(delta) ) }
 
+
+void database::adjust_balance(asset_id_type lottery_id, asset delta)
+{
+   if( delta.amount == 0 )
+      return;
+
+   auto& index = get_index_type<lottery_balance_index>().indices().get<by_owner>();
+   auto itr = index.find(lottery_id);
+   if(itr == index.end())
+   {
+      FC_ASSERT( delta.amount > 0, "Insufficient Balance: ${a}'s balance  is less than required ${r}",
+                 ("a",lottery_id)
+                 ("b","test")
+                 ("r",to_pretty_string(-delta)));
+      create<lottery_balance_object>([lottery_id,&delta](lottery_balance_object& b) {
+         b.lottery_id = lottery_id;
+         b.balance = asset(delta.amount, delta.asset_id);
+      });
+   } else {
+      if( delta.amount < 0 )
+         FC_ASSERT( itr->get_balance() >= -delta, "Insufficient Balance: ${a}'s balance of ${b} is less than required ${r}", ("a",lottery_id)("b",to_pretty_string(itr->get_balance()))("r",to_pretty_string(-delta)));
+      modify(*itr, [delta](lottery_balance_object& b) {
+         b.adjust_balance(delta);
+      });
+   }
+}
+
+
+void database::adjust_sweeps_vesting_balance(account_id_type account, int64_t delta)
+{
+   if( delta == 0 )
+      return;
+   
+   asset_id_type asset_id = get_global_properties().parameters.sweeps_distribution_asset();
+   
+   auto& index = get_index_type<sweeps_vesting_balance_index>().indices().get<by_owner>();
+   auto itr = index.find(account);
+   if(itr == index.end())
+   {
+      FC_ASSERT( delta > 0, "Insufficient Balance: ${a}'s balance of ${b} is less than required ${r}",
+                 ("a",account)
+                 ("b","test")
+                 ("r",-delta));
+      create<sweeps_vesting_balance_object>([account,&delta,&asset_id](sweeps_vesting_balance_object& b) {
+         b.owner = account;
+         b.asset_id = asset_id;
+         b.balance = delta;
+      });
+   } else {
+      if( delta < 0 )
+         FC_ASSERT( itr->get_balance() >= -delta, "Insufficient Balance: ${a}'s balance of ${b} is less than required ${r}", ("a",account)("b",itr->get_balance())("r",-delta));
+      modify(*itr, [&delta,&asset_id,this](sweeps_vesting_balance_object& b) {
+         b.adjust_balance( asset( delta, asset_id ) );
+         b.last_claim_date = head_block_time();
+      });
+   }
+}
+
 optional< vesting_balance_id_type > database::deposit_lazy_vesting(
    const optional< vesting_balance_id_type >& ovbid,
    share_type amount, uint32_t req_vesting_seconds,

libraries/chain/db_block.cpp

@@ -215,12 +215,15 @@ bool database::_push_block(const signed_block& new_block)
 
             // pop blocks until we hit the forked block
             while( head_block_id() != branches.second.back()->data.previous )
+            {
+               ilog( "popping block #${n} ${id}", ("n",head_block_num())("id",head_block_id()) );
                pop_block();
+            }
 
             // push all blocks on the new fork
             for( auto ritr = branches.first.rbegin(); ritr != branches.first.rend(); ++ritr )
             {
-                ilog( "pushing blocks from fork ${n} ${id}", ("n",(*ritr)->data.block_num())("id",(*ritr)->data.id()) );
+                ilog( "pushing block from fork #${n} ${id}", ("n",(*ritr)->data.block_num())("id",(*ritr)->id) );
                 optional<fc::exception> except;
                 try {
                    undo_database::session session = _undo_db.start_undo_session();
@@ -235,21 +238,27 @@ bool database::_push_block(const signed_block& new_block)
                    // remove the rest of branches.first from the fork_db, those blocks are invalid
                    while( ritr != branches.first.rend() )
                    {
-                      _fork_db.remove( (*ritr)->data.id() );
+                      ilog( "removing block from fork_db #${n} ${id}", ("n",(*ritr)->data.block_num())("id",(*ritr)->id) );
+                      _fork_db.remove( (*ritr)->id );
                       ++ritr;
                    }
                    _fork_db.set_head( branches.second.front() );
 
                    // pop all blocks from the bad fork
                    while( head_block_id() != branches.second.back()->data.previous )
-                      pop_block();
-
-                   // restore all blocks from the good fork
-                   for( auto ritr = branches.second.rbegin(); ritr != branches.second.rend(); ++ritr )
                    {
+                      ilog( "popping block #${n} ${id}", ("n",head_block_num())("id",head_block_id()) );
+                      pop_block();
+                   }
+
+                   ilog( "Switching back to fork: ${id}", ("id",branches.second.front()->data.id()) );
+                   // restore all blocks from the good fork
+                   for( auto ritr2 = branches.second.rbegin(); ritr2 != branches.second.rend(); ++ritr2 )
+                   {
+                      ilog( "pushing block #${n} ${id}", ("n",(*ritr2)->data.block_num())("id",(*ritr2)->id) );
                       auto session = _undo_db.start_undo_session();
-                      apply_block( (*ritr)->data, skip );
+                      apply_block( (*ritr2)->data, skip );
-                      _block_id_to_block.store( new_block.id(), (*ritr)->data );
+                      _block_id_to_block.store( (*ritr2)->id, (*ritr2)->data );
                       session.commit();
                    }
                    throw *except;
@@ -327,6 +336,8 @@ processed_transaction database::validate_transaction( const signed_transaction&
 
 processed_transaction database::push_proposal(const proposal_object& proposal)
 { try {
+   FC_ASSERT( _undo_db.size() < _undo_db.max_size(), "Undo database is full!" );
+
    transaction_evaluation_state eval_state(this);
    eval_state._is_proposed_trx = true;
 
@@ -466,10 +477,9 @@ signed_block database::_generate_block(
    pending_block.witness = witness_id;
    
    // Genesis witnesses start with a default initial secret
-   if( witness_obj.next_secret_hash == secret_hash_type::hash( secret_hash_type() ) )     
+   if( witness_obj.next_secret_hash == secret_hash_type::hash( secret_hash_type() ) ) {
          pending_block.previous_secret = secret_hash_type();
-   else       
+   } else {
-   {      
          secret_hash_type::encoder last_enc;
          fc::raw::pack( last_enc, block_signing_private_key );
          fc::raw::pack( last_enc, witness_obj.previous_secret );
@@ -490,7 +500,7 @@ signed_block database::_generate_block(
       FC_ASSERT( fc::raw::pack_size(pending_block) <= get_global_properties().parameters.maximum_block_size );
    }
 
-   push_block( pending_block, skip );
+   push_block( pending_block, skip | skip_transaction_signatures ); // skip authority check when pushing self-generated blocks
 
    return pending_block;
 } FC_CAPTURE_AND_RETHROW( (witness_id) ) }
@@ -507,7 +517,6 @@ void database::pop_block()
    GRAPHENE_ASSERT( head_block.valid(), pop_empty_chain, "there are no blocks to pop" );
 
    _fork_db.pop_block();
-   _block_id_to_block.remove( head_id );
    pop_undo();
 
    _popped_tx.insert( _popped_tx.begin(), head_block->transactions.begin(), head_block->transactions.end() );
@@ -588,6 +597,8 @@ void database::_apply_block( const signed_block& next_block )
 
    _current_block_num    = next_block_num;
    _current_trx_in_block = 0;
+   _current_op_in_trx    = 0;
+   _current_virtual_op   = 0;
 
    for( const auto& trx : next_block.transactions )
    {
@@ -597,13 +608,22 @@ void database::_apply_block( const signed_block& next_block )
        * for transactions when validating broadcast transactions or
        * when building a block.
        */
+
       apply_transaction( trx, skip );
+      // For real operations which are explicitly included in a transaction, virtual_op is 0.
+      // For VOPs derived directly from a real op,
+      //     use the real op's (block_num,trx_in_block,op_in_trx), virtual_op starts from 1.
+      // For VOPs created after processed all transactions,
+      //     trx_in_block = the_block.trsanctions.size(), virtual_op starts from 0.      
       ++_current_trx_in_block;
+      _current_op_in_trx  = 0;
+      _current_virtual_op = 0;   
    }
 
    if (global_props.parameters.witness_schedule_algorithm == GRAPHENE_WITNESS_SCHEDULED_ALGORITHM)
        update_witness_schedule(next_block);
-   update_global_dynamic_data(next_block);
+   const uint32_t missed = update_witness_missed_blocks( next_block );
+   update_global_dynamic_data( next_block, missed );
    update_signing_witness(signing_witness, next_block);
    update_last_irreversible_block();
 
@@ -611,6 +631,8 @@ void database::_apply_block( const signed_block& next_block )
    if( maint_needed )
       perform_chain_maintenance(next_block, global_props);
    
+   check_ending_lotteries();
+   
    create_block_summary(next_block);
    place_delayed_bets(); // must happen after update_global_dynamic_data() updates the time
    clear_expired_transactions();
@@ -651,6 +673,19 @@ processed_transaction database::apply_transaction(const signed_transaction& trx,
    return result;
 }
 
+class undo_size_restorer {
+   public:
+      undo_size_restorer( undo_database& db ) : _db( db ), old_max( db.max_size() ) {
+         _db.set_max_size( old_max * 2 );
+      }
+      ~undo_size_restorer() {
+         _db.set_max_size( old_max );
+      }
+   private:
+      undo_database& _db;
+      size_t         old_max;
+};
+
 processed_transaction database::_apply_transaction(const signed_transaction& trx)
 { try {
    uint32_t skip = get_node_properties().skip_flags;
@@ -660,9 +695,14 @@ processed_transaction database::_apply_transaction(const signed_transaction& trx
 
    auto& trx_idx = get_mutable_index_type<transaction_index>();
    const chain_id_type& chain_id = get_chain_id();
-   auto trx_id = trx.id();
+   transaction_id_type trx_id;
-   FC_ASSERT( (skip & skip_transaction_dupe_check) ||
+   
-              trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end() );
+   if( !(skip & skip_transaction_dupe_check) )
+   {
+      trx_id = trx.id();
+      FC_ASSERT( trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end() );
+   }
+
    transaction_evaluation_state eval_state(this);
    const chain_parameters& chain_parameters = get_global_properties().parameters;
    eval_state._trx = &trx;
@@ -696,7 +736,7 @@ processed_transaction database::_apply_transaction(const signed_transaction& trx
    //Insert transaction into unique transactions database.
    if( !(skip & skip_transaction_dupe_check) )
    {
-      create<transaction_object>([&](transaction_object& transaction) {
+      create<transaction_object>([&trx_id,&trx](transaction_object& transaction) {
          transaction.trx_id = trx_id;
          transaction.trx = trx;
       });
@@ -704,20 +744,23 @@ processed_transaction database::_apply_transaction(const signed_transaction& trx
 
    eval_state.operation_results.reserve(trx.operations.size());
 
+   const undo_size_restorer undo_guard( _undo_db );
    //Finally process the operations
    processed_transaction ptrx(trx);
    _current_op_in_trx = 0;
+   _current_virtual_op = 0;
    for( const auto& op : ptrx.operations )
    {
+      _current_virtual_op = 0;
       eval_state.operation_results.emplace_back(apply_operation(eval_state, op));
       ++_current_op_in_trx;
    }
    ptrx.operation_results = std::move(eval_state.operation_results);
 
    //Make sure the temp account has no non-zero balances
-   const auto& index = get_index_type<account_balance_index>().indices().get<by_account_asset>();
+   const auto& balances = get_index_type< primary_index< account_balance_index > >().get_secondary_index< balances_by_account_index >().get_account_balances( GRAPHENE_TEMP_ACCOUNT );
-   auto range = index.equal_range( boost::make_tuple( GRAPHENE_TEMP_ACCOUNT ) );
+   for( const auto b : balances )
-   std::for_each(range.first, range.second, [](const account_balance_object& b) { FC_ASSERT(b.balance == 0); });
+      FC_ASSERT(b.second->balance == 0);
 
    return ptrx;
 } FC_CAPTURE_AND_RETHROW( (trx) ) }
@@ -742,8 +785,9 @@ const witness_object& database::validate_block_header( uint32_t skip, const sign
    FC_ASSERT( head_block_time() < next_block.timestamp, "", ("head_block_time",head_block_time())("next",next_block.timestamp)("blocknum",next_block.block_num()) );
    const witness_object& witness = next_block.witness(*this);
 //DLN: TODO: Temporarily commented out to test shuffle vs RNG scheduling algorithm for witnesses, this was causing shuffle agorithm to fail during create_witness test. This should be re-enabled for RNG, and maybe for shuffle too, don't really know for sure.
-//   FC_ASSERT( secret_hash_type::hash( next_block.previous_secret ) == witness.next_secret_hash, "",        
+   if( next_block.timestamp > HARDFORK_SWEEPS_TIME )
-//              ("previous_secret", next_block.previous_secret)("next_secret_hash", witness.next_secret_hash)("null_secret_hash", secret_hash_type::hash( secret_hash_type())));
+      FC_ASSERT( secret_hash_type::hash( next_block.previous_secret ) == witness.next_secret_hash, "",
+               ( "previous_secret", next_block.previous_secret )( "next_secret_hash", witness.next_secret_hash ) );
 
    if( !(skip&skip_witness_signature) ) 
       FC_ASSERT( next_block.validate_signee( witness.signing_key ) );

libraries/chain/db_debug.cpp

@@ -118,10 +118,10 @@ void debug_apply_update( database& db, const fc::variant_object& vo )
    auto it_id = vo.find("id");
    FC_ASSERT( it_id != vo.end() );
 
-   from_variant( it_id->value(), oid );
+   from_variant( it_id->value(), oid, GRAPHENE_MAX_NESTED_OBJECTS );
    action = ( vo.size() == 1 ) ? db_action_delete : db_action_write;
 
-   from_variant( vo["id"], oid );
+   from_variant( vo["id"], oid, GRAPHENE_MAX_NESTED_OBJECTS );
    if( vo.size() == 1 )
       action = db_action_delete;
    auto it_action = vo.find("_action" );
@@ -143,25 +143,19 @@ void debug_apply_update( database& db, const fc::variant_object& vo )
    switch( action )
    {
       case db_action_create:
-         /*
-         idx.create( [&]( object& obj )
-         {
-            idx.object_from_variant( vo, obj );
-         } );
-         */
          FC_ASSERT( false );
          break;
       case db_action_write:
          db.modify( db.get_object( oid ), [&]( object& obj )
          {
             idx.object_default( obj );
-            idx.object_from_variant( vo, obj );
+            idx.object_from_variant( vo, obj, GRAPHENE_MAX_NESTED_OBJECTS );
          } );
          break;
       case db_action_update:
          db.modify( db.get_object( oid ), [&]( object& obj )
          {
-            idx.object_from_variant( vo, obj );
+            idx.object_from_variant( vo, obj, GRAPHENE_MAX_NESTED_OBJECTS );
          } );
          break;
       case db_action_delete:

libraries/chain/db_getter.cpp

@@ -30,6 +30,9 @@
 
 #include <fc/smart_ref_impl.hpp>
 
+#include <ctime>
+#include <algorithm>
+
 namespace graphene { namespace chain {
 
 const asset_object& database::get_core_asset() const
@@ -97,5 +100,45 @@ uint32_t database::last_non_undoable_block_num() const
    return head_block_num() - _undo_db.size();
 }
 
+std::vector<uint32_t> database::get_seeds(asset_id_type for_asset, uint8_t count_winners) const
+{
+   FC_ASSERT( count_winners <= 64 );
+   std::string salted_string = std::string(_random_number_generator._seed) + std::to_string(for_asset.instance.value);
+   uint32_t* seeds = (uint32_t*)(fc::sha256::hash(salted_string)._hash);
+
+   std::vector<uint32_t> result;
+   result.reserve(64);
+
+   for( int s = 0; s < 8; ++s ) {
+      uint32_t* sub_seeds = ( uint32_t* ) fc::sha256::hash( std::to_string( seeds[s] ) + std::to_string( for_asset.instance.value ) )._hash;
+      for( int ss = 0; ss < 8; ++ss ) {
+         result.push_back(sub_seeds[ss]);
+      }
+   }
+   return result;
+}
+
+const std::vector<uint32_t> database::get_winner_numbers( asset_id_type for_asset, uint32_t count_members, uint8_t count_winners ) const
+{
+   std::vector<uint32_t> result;
+   if( count_members < count_winners ) count_winners = count_members;
+   if( count_winners == 0 ) return result;
+   result.reserve(count_winners);
+
+   auto seeds = get_seeds(for_asset, count_winners);
+
+   for (auto current_seed = seeds.begin(); current_seed != seeds.end(); ++current_seed) {
+      uint8_t winner_num = *current_seed % count_members;
+      while( std::find(result.begin(), result.end(), winner_num) != result.end() ) {
+         *current_seed = (*current_seed * 1103515245 + 12345) / 65536; //using gcc's consts for pseudorandom
+         winner_num = *current_seed % count_members;
+      }
+      result.push_back(winner_num);
+      if (result.size() >= count_winners) break;
+   }
+   
+   FC_ASSERT(result.size() == count_winners);
+   return result;
+}
 
 } }

libraries/chain/db_init.cpp

@@ -59,6 +59,7 @@
 
 #include <graphene/chain/account_evaluator.hpp>
 #include <graphene/chain/asset_evaluator.hpp>
+#include <graphene/chain/lottery_evaluator.hpp>
 #include <graphene/chain/assert_evaluator.hpp>
 #include <graphene/chain/balance_evaluator.hpp>
 #include <graphene/chain/committee_member_evaluator.hpp>
@@ -237,6 +238,11 @@ void database::initialize_evaluators()
    register_evaluator<tournament_join_evaluator>();
    register_evaluator<game_move_evaluator>();
    register_evaluator<tournament_leave_evaluator>();
+   register_evaluator<lottery_asset_create_evaluator>();
+   register_evaluator<ticket_purchase_evaluator>();
+   register_evaluator<lottery_reward_evaluator>();
+   register_evaluator<lottery_end_evaluator>();
+   register_evaluator<sweeps_vesting_claim_evaluator>();
 }
 
 void database::initialize_indexes()
@@ -245,15 +251,15 @@ void database::initialize_indexes()
    _undo_db.set_max_size( GRAPHENE_MIN_UNDO_HISTORY );
 
    //Protocol object indexes
-   add_index< primary_index<asset_index> >();
+   add_index< primary_index<asset_index, 13> >(); // 8192 assets per chunk
    add_index< primary_index<force_settlement_index> >();
 
-   auto acnt_index = add_index< primary_index<account_index> >();
+   auto acnt_index = add_index< primary_index<account_index, 20> >(); // ~1 million accounts per chunk
    acnt_index->add_secondary_index<account_member_index>();
    acnt_index->add_secondary_index<account_referrer_index>();
 
-   add_index< primary_index<committee_member_index> >();
+   add_index< primary_index<committee_member_index, 8> >(); // 256 members per chunk
-   add_index< primary_index<witness_index> >();
+   add_index< primary_index<witness_index, 10> >(); // 1024 witnesses per chunk
    add_index< primary_index<limit_order_index > >();
    add_index< primary_index<call_order_index > >();
 
@@ -281,8 +287,11 @@ void database::initialize_indexes()
 
    //Implementation object indexes
    add_index< primary_index<transaction_index                             > >();
-   add_index< primary_index<account_balance_index                         > >();
+
-   add_index< primary_index<asset_bitasset_data_index                     > >();
+   auto bal_idx = add_index< primary_index<account_balance_index          > >();
+   bal_idx->add_secondary_index<balances_by_account_index>();
+
+   add_index< primary_index<asset_bitasset_data_index,                 13 > >(); // 8192
    add_index< primary_index<asset_dividend_data_object_index              > >();
    add_index< primary_index<simple_index<global_property_object          >> >();
    add_index< primary_index<simple_index<dynamic_global_property_object  >> >();
@@ -301,6 +310,10 @@ void database::initialize_indexes()
    //add_index< primary_index<distributed_dividend_balance_object_index > >();
    add_index< primary_index<pending_dividend_payout_balance_for_holder_object_index > >();
    add_index< primary_index<total_distributed_dividend_balance_object_index > >();
+   
+   add_index< primary_index<lottery_balance_index                         > >();
+   add_index< primary_index<sweeps_vesting_balance_index                  > >();
+
 }
 
 void database::init_genesis(const genesis_state_type& genesis_state)
@@ -729,7 +742,7 @@ void database::init_genesis(const genesis_state_type& genesis_state)
                vbo.owner = get_account_id(account.name);
                vbo.balance = asset(vesting_balance.amount, get_asset_id(vesting_balance.asset_symbol));
                if (vesting_balance.policy_type == "linear") {
-                  auto initial_linear_vesting_policy = vesting_balance.policy.as<genesis_state_type::initial_bts_account_type::initial_linear_vesting_policy>();
+                  auto initial_linear_vesting_policy = vesting_balance.policy.as<genesis_state_type::initial_bts_account_type::initial_linear_vesting_policy>( 20 );
                   linear_vesting_policy new_vesting_policy;
                   new_vesting_policy.begin_timestamp = initial_linear_vesting_policy.begin_timestamp;
                   new_vesting_policy.vesting_cliff_seconds = initial_linear_vesting_policy.vesting_cliff_seconds;
@@ -737,7 +750,7 @@ void database::init_genesis(const genesis_state_type& genesis_state)
                   new_vesting_policy.begin_balance = initial_linear_vesting_policy.begin_balance;
                   vbo.policy = new_vesting_policy;
                } else if (vesting_balance.policy_type == "cdd") {
-                  auto initial_cdd_vesting_policy = vesting_balance.policy.as<genesis_state_type::initial_bts_account_type::initial_cdd_vesting_policy>();
+                  auto initial_cdd_vesting_policy = vesting_balance.policy.as<genesis_state_type::initial_bts_account_type::initial_cdd_vesting_policy>( 20 );
                   cdd_vesting_policy new_vesting_policy;
                   new_vesting_policy.vesting_seconds = initial_cdd_vesting_policy.vesting_seconds;
                   new_vesting_policy.coin_seconds_earned = initial_cdd_vesting_policy.coin_seconds_earned;
@@ -852,7 +865,7 @@ void database::init_genesis(const genesis_state_type& genesis_state)
    std::for_each(genesis_state.initial_witness_candidates.begin(), genesis_state.initial_witness_candidates.end(),
                  [&](const genesis_state_type::initial_witness_type& witness) {
       witness_create_operation op;
-      op.initial_secret = secret_hash_type::hash(secret_hash_type());
+      op.initial_secret = secret_hash_type();
       op.witness_account = get_account_id(witness.owner_name);
       op.block_signing_key = witness.block_signing_key;
       apply_operation(genesis_eval_state, op);

libraries/chain/db_maint.cpp

@@ -621,7 +621,7 @@ void distribute_fba_balances( database& db )
 void create_buyback_orders( database& db )
 {
    const auto& bbo_idx = db.get_index_type< buyback_index >().indices().get<by_id>();
-   const auto& bal_idx = db.get_index_type< account_balance_index >().indices().get< by_account_asset >();
+   const auto& bal_idx = db.get_index_type< primary_index< account_balance_index > >().get_secondary_index< balances_by_account_index >();
 
    for( const buyback_object& bbo : bbo_idx )
    {
@@ -629,7 +629,6 @@ void create_buyback_orders( database& db )
       assert( asset_to_buy.buyback_account.valid() );
 
       const account_object& buyback_account = (*(asset_to_buy.buyback_account))(db);
-      asset_id_type next_asset = asset_id_type();
 
       if( !buyback_account.allowed_assets.valid() )
       {
@@ -637,16 +636,11 @@ void create_buyback_orders( database& db )
          continue;
       }
 
-      while( true )
+      for( const auto& entry : bal_idx.get_account_balances( buyback_account.id ) )
       {
-         auto it = bal_idx.lower_bound( boost::make_tuple( buyback_account.id, next_asset ) );
+         const auto* it = entry.second;
-         if( it == bal_idx.end() )
-            break;
-         if( it->owner != buyback_account.id )
-            break;
          asset_id_type asset_to_sell = it->asset_type;
          share_type amount_to_sell = it->balance;
-         next_asset = asset_to_sell + 1;
          if( asset_to_sell == asset_to_buy.id )
             continue;
          if( amount_to_sell == 0 )
@@ -725,120 +719,6 @@ void deprecate_annual_members( database& db )
    return;
 }
 
-double database::calculate_vesting_factor(const account_object& stake_account)
-{
-   // get last time voted form stats
-   const auto &stats = stake_account.statistics(*this);
-   fc::time_point_sec last_date_voted = stats.last_vote_time;
-
-   // get global data related to gpos
-   const auto &gpo = this->get_global_properties();
-   const auto vesting_period = gpo.parameters.gpos_period();
-   const auto vesting_subperiod = gpo.parameters.gpos_subperiod();
-   const auto period_start = fc::time_point_sec(gpo.parameters.gpos_period_start());
-
-   //  variables needed
-   const fc::time_point_sec period_end = period_start + vesting_period;
-   const auto number_of_subperiods = vesting_period / vesting_subperiod;
-   const auto now = this->head_block_time();
-   double vesting_factor;
-   auto seconds_since_period_start = now.sec_since_epoch() - period_start.sec_since_epoch();
-
-   FC_ASSERT(period_start <= now && now <= period_end);
-
-   // get in what sub period we are
-   uint32_t current_subperiod = 0;
-   std::list<uint32_t> period_list(number_of_subperiods);
-   std::iota(period_list.begin(), period_list.end(), 1);
-
-   std::for_each(period_list.begin(), period_list.end(),[&](uint32_t period) {
-      if(seconds_since_period_start >= vesting_subperiod * (period - 1) &&
-            seconds_since_period_start < vesting_subperiod * period)
-         current_subperiod = period;
-   });
-
-   if(current_subperiod == 0 || current_subperiod > number_of_subperiods) return 0;
-   if(last_date_voted < period_start) return 0;
-
-   double numerator = number_of_subperiods;
-
-   if(current_subperiod > 1) {
-      std::list<uint32_t> subperiod_list(current_subperiod - 1);
-      std::iota(subperiod_list.begin(), subperiod_list.end(), 2);
-      subperiod_list.reverse();
-
-      for(auto subperiod: subperiod_list)
-      {
-         numerator--;
-
-         auto last_period_start = period_start + fc::seconds(vesting_subperiod * (subperiod - 1));
-         auto last_period_end = period_start + fc::seconds(vesting_subperiod * (subperiod));
-
-         if (last_date_voted > last_period_start && last_date_voted <= last_period_end) {
-            numerator++;
-            break;
-         }
-      }
-   }
-   vesting_factor = numerator / number_of_subperiods;
-   return vesting_factor;
-}
-
-share_type credit_account(database& db, const account_id_type owner_id, const std::string owner_name,
-                          share_type remaining_amount_to_distribute,
-                          const share_type shares_to_credit, const asset_id_type payout_asset_type,
-                          const pending_dividend_payout_balance_for_holder_object_index& pending_payout_balance_index,
-                          const asset_id_type dividend_id) {
-
-   //wdump((delta_balance.value)(holder_balance)(total_balance_of_dividend_asset));
-   if (shares_to_credit.value) {
-
-      remaining_amount_to_distribute -= shares_to_credit;
-
-      dlog("Crediting account ${account} with ${amount}",
-           ("account", owner_name)
-                 ("amount", asset(shares_to_credit, payout_asset_type)));
-      auto pending_payout_iter =
-            pending_payout_balance_index.indices().get<by_dividend_payout_account>().find(
-                  boost::make_tuple(dividend_id, payout_asset_type,
-                                    owner_id));
-      if (pending_payout_iter ==
-          pending_payout_balance_index.indices().get<by_dividend_payout_account>().end())
-         db.create<pending_dividend_payout_balance_for_holder_object>(
-               [&](pending_dividend_payout_balance_for_holder_object &obj) {
-                  obj.owner = owner_id;
-                  obj.dividend_holder_asset_type = dividend_id;
-                  obj.dividend_payout_asset_type = payout_asset_type;
-                  obj.pending_balance = shares_to_credit;
-               });
-      else
-         db.modify(*pending_payout_iter,
-                   [&](pending_dividend_payout_balance_for_holder_object &pending_balance) {
-                      pending_balance.pending_balance += shares_to_credit;
-                   });
-   }
-   return remaining_amount_to_distribute;
-}
-
-void rolling_period_start(database& db)
-{
-   if(db.head_block_time() >= HARDFORK_GPOS_TIME)
-   {
-      auto gpo = db.get_global_properties();
-      auto period_start = db.get_global_properties().parameters.gpos_period_start();
-      auto vesting_period = db.get_global_properties().parameters.gpos_period();
-
-      auto now = db.head_block_time();
-      if(now.sec_since_epoch() > (period_start + vesting_period))
-      {
-         // roll
-         db.modify(db.get_global_properties(), [now](global_property_object& p) {
-            p.parameters.extensions.value.gpos_period_start =  now.sec_since_epoch();
-         });
-      }
-   }
-}
-
 // Schedules payouts from a dividend distribution account to the current holders of the
 // dividend-paying asset.  This takes any deposits made to the dividend distribution account
 // since the last time it was called, and distributes them to the current owners of the
@@ -854,8 +734,10 @@ void schedule_pending_dividend_balances(database& db,
 { try {
    dlog("Processing dividend payments for dividend holder asset type ${holder_asset} at time ${t}",
         ("holder_asset", dividend_holder_asset_obj.symbol)("t", db.head_block_time()));
+   auto balance_by_acc_index = db.get_index_type< primary_index< account_balance_index > >().get_secondary_index< balances_by_account_index >();
    auto current_distribution_account_balance_range = 
-      balance_index.indices().get<by_account_asset>().equal_range(boost::make_tuple(dividend_data.dividend_distribution_account));
+      //balance_index.indices().get<by_account_asset>().equal_range(boost::make_tuple(dividend_data.dividend_distribution_account));
+      balance_by_acc_index.get_account_balances(dividend_data.dividend_distribution_account);
    auto previous_distribution_account_balance_range =
       distributed_dividend_balance_index.indices().get<by_dividend_payout_asset>().equal_range(boost::make_tuple(dividend_holder_asset_obj.id));
    // the current range is now all current balances for the distribution account, sorted by asset_type
@@ -868,41 +750,34 @@ void schedule_pending_dividend_balances(database& db,
       balance_index.indices().get<by_asset_balance>().lower_bound(boost::make_tuple(dividend_holder_asset_obj.id));
    auto holder_balances_end =
       balance_index.indices().get<by_asset_balance>().upper_bound(boost::make_tuple(dividend_holder_asset_obj.id, share_type()));
+   uint32_t holder_account_count = std::distance(holder_balances_begin, holder_balances_end);
    uint64_t distribution_base_fee = gpo.parameters.current_fees->get<asset_dividend_distribution_operation>().distribution_base_fee;
    uint32_t distribution_fee_per_holder = gpo.parameters.current_fees->get<asset_dividend_distribution_operation>().distribution_fee_per_holder;
+   // the fee, in BTS, for distributing each asset in the account
+   uint64_t total_fee_per_asset_in_core = distribution_base_fee + holder_account_count * (uint64_t)distribution_fee_per_holder;
 
    std::map<account_id_type, share_type> vesting_amounts;
-
-   auto balance_type = vesting_balance_type::unspecified;
-   if(db.head_block_time() >= HARDFORK_GPOS_TIME)
-      balance_type = vesting_balance_type::gpos;
-
-   uint32_t holder_account_count = 0;
 #ifdef USE_VESTING_OBJECT_BY_ASSET_BALANCE_INDEX
    // get only once a collection of accounts that hold nonzero vesting balances of the dividend asset
    auto vesting_balances_begin =
-      vesting_index.indices().get<by_asset_balance>().lower_bound(boost::make_tuple(dividend_holder_asset_obj.id, balance_type));
+      vesting_index.indices().get<by_asset_balance>().lower_bound(boost::make_tuple(dividend_holder_asset_obj.id));
    auto vesting_balances_end =
-      vesting_index.indices().get<by_asset_balance>().upper_bound(boost::make_tuple(dividend_holder_asset_obj.id, balance_type, share_type()));
+      vesting_index.indices().get<by_asset_balance>().upper_bound(boost::make_tuple(dividend_holder_asset_obj.id, share_type()));
-
    for (const vesting_balance_object& vesting_balance_obj : boost::make_iterator_range(vesting_balances_begin, vesting_balances_end))
    {
         vesting_amounts[vesting_balance_obj.owner] += vesting_balance_obj.balance.amount;
-        ++holder_account_count;
+        //dlog("Vesting balance for account: ${owner}, amount: ${amount}",
-        dlog("Vesting balance for account: ${owner}, amount: ${amount}",
+        //     ("owner", vesting_balance_obj.owner(db).name)
-             ("owner", vesting_balance_obj.owner(db).name)
+        //     ("amount", vesting_balance_obj.balance.amount));
-             ("amount", vesting_balance_obj.balance.amount));
    }
 #else
    // get only once a collection of accounts that hold nonzero vesting balances of the dividend asset
    const auto& vesting_balances = vesting_index.indices().get<by_id>();
    for (const vesting_balance_object& vesting_balance_obj : vesting_balances)
    {
-        if (vesting_balance_obj.balance.asset_id == dividend_holder_asset_obj.id && vesting_balance_obj.balance.amount &&
+        if (vesting_balance_obj.balance.asset_id == dividend_holder_asset_obj.id && vesting_balance_obj.balance.amount)
-        vesting_balance_object.balance_type == balance_type)
         {
             vesting_amounts[vesting_balance_obj.owner] += vesting_balance_obj.balance.amount;
-            ++gpos_holder_account_count;
             dlog("Vesting balance for account: ${owner}, amount: ${amount}",
                  ("owner", vesting_balance_obj.owner(db).name)
                  ("amount", vesting_balance_obj.balance.amount));
@@ -910,40 +785,26 @@ void schedule_pending_dividend_balances(database& db,
    }
 #endif
 
-   if(db.head_block_time() < HARDFORK_GPOS_TIME)
+   auto current_distribution_account_balance_iter = current_distribution_account_balance_range.begin();
-      holder_account_count = std::distance(holder_balances_begin, holder_balances_end);
-   // the fee, in BTS, for distributing each asset in the account
-   uint64_t total_fee_per_asset_in_core = distribution_base_fee + holder_account_count * (uint64_t)distribution_fee_per_holder;
-
-   auto current_distribution_account_balance_iter = current_distribution_account_balance_range.first;
    auto previous_distribution_account_balance_iter = previous_distribution_account_balance_range.first;
    dlog("Current balances in distribution account: ${current}, Previous balances: ${previous}",
-        ("current", std::distance(current_distribution_account_balance_range.first, current_distribution_account_balance_range.second))
+        ("current", (int64_t)std::distance(current_distribution_account_balance_range.begin(), current_distribution_account_balance_range.end()))
-        ("previous", std::distance(previous_distribution_account_balance_range.first, previous_distribution_account_balance_range.second)));
+        ("previous", (int64_t)std::distance(previous_distribution_account_balance_range.first, previous_distribution_account_balance_range.second)));
 
    // when we pay out the dividends to the holders, we need to know the total balance of the dividend asset in all
    // accounts other than the distribution account (it would be silly to distribute dividends back to 
    // the distribution account)
    share_type total_balance_of_dividend_asset;
-   if(db.head_block_time() >= HARDFORK_GPOS_TIME && dividend_holder_asset_obj.symbol == GRAPHENE_SYMBOL) { // only core
+   for (const account_balance_object& holder_balance_object : boost::make_iterator_range(holder_balances_begin, holder_balances_end))
-      for (const vesting_balance_object &holder_balance_object : boost::make_iterator_range(vesting_balances_begin,
+      if (holder_balance_object.owner != dividend_data.dividend_distribution_account)
-                                                                                            vesting_balances_end))
+      {
-         if (holder_balance_object.owner != dividend_data.dividend_distribution_account) {
-            total_balance_of_dividend_asset += holder_balance_object.balance.amount;
-         }
-   }
-   else {
-      for (const account_balance_object &holder_balance_object : boost::make_iterator_range(holder_balances_begin,
-                                                                                            holder_balances_end))
-         if (holder_balance_object.owner != dividend_data.dividend_distribution_account) {
          total_balance_of_dividend_asset += holder_balance_object.balance;
          auto itr = vesting_amounts.find(holder_balance_object.owner);
          if (itr != vesting_amounts.end())
              total_balance_of_dividend_asset += itr->second;
       }
-   }
    // loop through all of the assets currently or previously held in the distribution account
-   while (current_distribution_account_balance_iter != current_distribution_account_balance_range.second ||
+   while (current_distribution_account_balance_iter != current_distribution_account_balance_range.end() ||
           previous_distribution_account_balance_iter != previous_distribution_account_balance_range.second)
    {
       try
@@ -954,15 +815,15 @@ void schedule_pending_dividend_balances(database& db,
          asset_id_type payout_asset_type;
 
          if (previous_distribution_account_balance_iter == previous_distribution_account_balance_range.second || 
-             current_distribution_account_balance_iter->asset_type < previous_distribution_account_balance_iter->dividend_payout_asset_type)
+             current_distribution_account_balance_iter->second->asset_type < previous_distribution_account_balance_iter->dividend_payout_asset_type)
          {
             // there are no more previous balances or there is no previous balance for this particular asset type
-            payout_asset_type = current_distribution_account_balance_iter->asset_type;
+            payout_asset_type = current_distribution_account_balance_iter->second->asset_type;
-            current_balance = current_distribution_account_balance_iter->balance;
+            current_balance = current_distribution_account_balance_iter->second->balance;
             idump((payout_asset_type)(current_balance));
          }
-         else if (current_distribution_account_balance_iter == current_distribution_account_balance_range.second || 
+         else if (current_distribution_account_balance_iter == current_distribution_account_balance_range.end() || 
-                  previous_distribution_account_balance_iter->dividend_payout_asset_type < current_distribution_account_balance_iter->asset_type)
+                  previous_distribution_account_balance_iter->dividend_payout_asset_type < current_distribution_account_balance_iter->second->asset_type)
          {
             // there are no more current balances or there is no current balance for this particular previous asset type
             payout_asset_type = previous_distribution_account_balance_iter->dividend_payout_asset_type;
@@ -972,8 +833,8 @@ void schedule_pending_dividend_balances(database& db,
          else
          {
             // we have both a previous and a current balance for this asset type
-            payout_asset_type = current_distribution_account_balance_iter->asset_type;
+            payout_asset_type = current_distribution_account_balance_iter->second->asset_type;
-            current_balance = current_distribution_account_balance_iter->balance;
+            current_balance = current_distribution_account_balance_iter->second->balance;
             previous_balance = previous_distribution_account_balance_iter->balance_at_last_maintenance_interval;
             idump((payout_asset_type)(current_balance)(previous_balance));
          }
@@ -1065,45 +926,9 @@ void schedule_pending_dividend_balances(database& db,
                     ("total", total_balance_of_dividend_asset));
                share_type remaining_amount_to_distribute = delta_balance;
 
-               if(db.head_block_time() >= HARDFORK_GPOS_TIME && dividend_holder_asset_obj.symbol == GRAPHENE_SYMBOL) { // core only
                // credit each account with their portion, don't send any back to the dividend distribution account
-                  for (const vesting_balance_object &holder_balance_object : boost::make_iterator_range(
+               for (const account_balance_object& holder_balance_object : boost::make_iterator_range(holder_balances_begin, holder_balances_end))
-                        vesting_balances_begin, vesting_balances_end)) {
+               {
-                     if (holder_balance_object.owner == dividend_data.dividend_distribution_account) continue;
-
-                     auto vesting_factor = db.calculate_vesting_factor(holder_balance_object.owner(db));
-
-                     auto holder_balance = holder_balance_object.balance;
-
-                     fc::uint128_t amount_to_credit(delta_balance.value);
-                     amount_to_credit *= holder_balance.amount.value;
-                     amount_to_credit /= total_balance_of_dividend_asset.value;
-                     share_type full_shares_to_credit((int64_t) amount_to_credit.to_uint64());
-                     share_type shares_to_credit = (uint64_t) floor(full_shares_to_credit.value * vesting_factor);
-
-                     if (shares_to_credit < full_shares_to_credit) {
-                        // Todo: sending results of decay to committee account, need to change to specified account
-                        dlog("Crediting committee_account with ${amount}",
-                             ("amount", asset(full_shares_to_credit - shares_to_credit, payout_asset_type)));
-                        db.adjust_balance(dividend_data.dividend_distribution_account,
-                                          -(full_shares_to_credit - shares_to_credit));
-                        db.adjust_balance(account_id_type(0), full_shares_to_credit - shares_to_credit);
-                     }
-
-                     remaining_amount_to_distribute = credit_account(db,
-                                                                     holder_balance_object.owner,
-                                                                     holder_balance_object.owner(db).name,
-                                                                     remaining_amount_to_distribute,
-                                                                     shares_to_credit,
-                                                                     payout_asset_type,
-                                                                     pending_payout_balance_index,
-                                                                     dividend_holder_asset_obj.id);
-                  }
-               }
-               else {
-                  // credit each account with their portion, don't send any back to the dividend distribution account
-                  for (const account_balance_object &holder_balance_object : boost::make_iterator_range(
-                        holder_balances_begin, holder_balances_end)) {
                   if (holder_balance_object.owner == dividend_data.dividend_distribution_account) continue;
 
                   auto holder_balance = holder_balance_object.balance;
@@ -1115,18 +940,32 @@ void schedule_pending_dividend_balances(database& db,
                   fc::uint128_t amount_to_credit(delta_balance.value);
                   amount_to_credit *= holder_balance.value;
                   amount_to_credit /= total_balance_of_dividend_asset.value;
-                     share_type shares_to_credit((int64_t) amount_to_credit.to_uint64());
+                  share_type shares_to_credit((int64_t)amount_to_credit.to_uint64());
+                  if (shares_to_credit.value)
+                  {
+                     wdump((delta_balance.value)(holder_balance)(total_balance_of_dividend_asset));
 
-                     remaining_amount_to_distribute = credit_account(db,
+                     remaining_amount_to_distribute -= shares_to_credit;
-                                                                     holder_balance_object.owner,
+
-                                                                     holder_balance_object.owner(db).name,
+                     dlog("Crediting account ${account} with ${amount}", 
-                                                                     remaining_amount_to_distribute,
+                          ("account", holder_balance_object.owner(db).name)
-                                                                     shares_to_credit,
+                          ("amount", asset(shares_to_credit, payout_asset_type)));
-                                                                     payout_asset_type,
+                     auto pending_payout_iter = 
-                                                                     pending_payout_balance_index,
+                        pending_payout_balance_index.indices().get<by_dividend_payout_account>().find(boost::make_tuple(dividend_holder_asset_obj.id, payout_asset_type, holder_balance_object.owner));
-                                                                     dividend_holder_asset_obj.id);
+                     if (pending_payout_iter == pending_payout_balance_index.indices().get<by_dividend_payout_account>().end())
+                        db.create<pending_dividend_payout_balance_for_holder_object>( [&]( pending_dividend_payout_balance_for_holder_object& obj ){
+                           obj.owner = holder_balance_object.owner;
+                           obj.dividend_holder_asset_type = dividend_holder_asset_obj.id;
+                           obj.dividend_payout_asset_type = payout_asset_type;
+                           obj.pending_balance = shares_to_credit;
+                        });
+                     else
+                        db.modify(*pending_payout_iter, [&]( pending_dividend_payout_balance_for_holder_object& pending_balance ){
+                           pending_balance.pending_balance += shares_to_credit;
+                        });
                   }
                }
+
                for (const auto& pending_payout : pending_payout_balance_index.indices())
                   if (pending_payout.pending_balance.value)
                       dlog("Pending payout: ${account_name}   ->   ${amount}",
@@ -1200,10 +1039,10 @@ void schedule_pending_dividend_balances(database& db,
 
       // iterate
       if (previous_distribution_account_balance_iter == previous_distribution_account_balance_range.second || 
-          current_distribution_account_balance_iter->asset_type < previous_distribution_account_balance_iter->dividend_payout_asset_type)
+          current_distribution_account_balance_iter->second->asset_type < previous_distribution_account_balance_iter->dividend_payout_asset_type)
          ++current_distribution_account_balance_iter;
-      else if (current_distribution_account_balance_iter == current_distribution_account_balance_range.second || 
+      else if (current_distribution_account_balance_iter == current_distribution_account_balance_range.end() || 
-               previous_distribution_account_balance_iter->dividend_payout_asset_type < current_distribution_account_balance_iter->asset_type)
+               previous_distribution_account_balance_iter->dividend_payout_asset_type < current_distribution_account_balance_iter->second->asset_type)
          ++previous_distribution_account_balance_iter;
       else
       {
@@ -1223,6 +1062,7 @@ void process_dividend_assets(database& db)
    ilog("In process_dividend_assets time ${time}", ("time", db.head_block_time()));
 
    const account_balance_index& balance_index = db.get_index_type<account_balance_index>();
+   //const auto& balance_index = db.get_index_type< primary_index< account_balance_index > >().get_secondary_index< balances_by_account_index >();
    const vesting_balance_index& vbalance_index = db.get_index_type<vesting_balance_index>();
    const total_distributed_dividend_balance_object_index& distributed_dividend_balance_index = db.get_index_type<total_distributed_dividend_balance_object_index>();
    const pending_dividend_payout_balance_for_holder_object_index& pending_payout_balance_index = db.get_index_type<pending_dividend_payout_balance_for_holder_object_index>();
@@ -1247,10 +1087,10 @@ void process_dividend_assets(database& db)
                     ("holder_asset", dividend_holder_asset_obj.symbol));
 #ifndef NDEBUG
                // dump balances before the payouts for debugging
-               const auto& balance_idx = db.get_index_type<account_balance_index>().indices().get<by_account_asset>();
+               const auto& balance_index = db.get_index_type< primary_index< account_balance_index > >();
-               auto holder_account_balance_range = balance_idx.equal_range(boost::make_tuple(dividend_data.dividend_distribution_account));
+               const auto& balances = balance_index.get_secondary_index< balances_by_account_index >().get_account_balances( dividend_data.dividend_distribution_account );
-               for (const account_balance_object& holder_balance_object : boost::make_iterator_range(holder_account_balance_range.first, holder_account_balance_range.second))
+               for( const auto balance : balances )
-                  ilog("  Current balance: ${asset}", ("asset", asset(holder_balance_object.balance, holder_balance_object.asset_type)));
+                  ilog("  Current balance: ${asset}", ("asset", asset(balance.second->balance, balance.second->asset_type)));
 #endif
 
                // when we do the payouts, we first increase the balances in all of the receiving accounts
@@ -1386,8 +1226,6 @@ void database::perform_chain_maintenance(const signed_block& next_block, const g
    distribute_fba_balances(*this);
    create_buyback_orders(*this);
 
-   rolling_period_start(*this);
-
    process_dividend_assets(*this);
 
    struct vote_tally_helper {
@@ -1403,28 +1241,24 @@ void database::perform_chain_maintenance(const signed_block& next_block, const g
          d._committee_count_histogram_buffer.resize(props.parameters.maximum_committee_count / 2 + 1);
          d._total_voting_stake = 0;
 
-         auto balance_type = vesting_balance_type::unspecified;
-         if(d.head_block_time() >= HARDFORK_GPOS_TIME)
-            balance_type = vesting_balance_type::gpos;
-
          const vesting_balance_index& vesting_index = d.get_index_type<vesting_balance_index>();
 #ifdef USE_VESTING_OBJECT_BY_ASSET_BALANCE_INDEX
          auto vesting_balances_begin =
-              vesting_index.indices().get<by_asset_balance>().lower_bound(boost::make_tuple(asset_id_type(), balance_type));
+              vesting_index.indices().get<by_asset_balance>().lower_bound(boost::make_tuple(asset_id_type()));
          auto vesting_balances_end =
-              vesting_index.indices().get<by_asset_balance>().upper_bound(boost::make_tuple(asset_id_type(), balance_type, share_type()));
+              vesting_index.indices().get<by_asset_balance>().upper_bound(boost::make_tuple(asset_id_type(), share_type()));
          for (const vesting_balance_object& vesting_balance_obj : boost::make_iterator_range(vesting_balances_begin, vesting_balances_end))
          {
             vesting_amounts[vesting_balance_obj.owner] += vesting_balance_obj.balance.amount;
-            dlog("Vesting balance for account: ${owner}, amount: ${amount}",
+            //dlog("Vesting balance for account: ${owner}, amount: ${amount}",
-                 ("owner", vesting_balance_obj.owner(d).name)
+            //     ("owner", vesting_balance_obj.owner(d).name)
-                 ("amount", vesting_balance_obj.balance.amount));
+            //     ("amount", vesting_balance_obj.balance.amount));
          }
 #else
          const auto& vesting_balances = vesting_index.indices().get<by_id>();
          for (const vesting_balance_object& vesting_balance_obj : vesting_balances)
          {
-            if (vesting_balance_obj.balance.asset_id == asset_id_type() && vesting_balance_obj.balance.amount && vesting_balance_obj.balance_type == balance_type)
+            if (vesting_balance_obj.balance.asset_id == asset_id_type() && vesting_balance_obj.balance.amount)
             {
                 vesting_amounts[vesting_balance_obj.owner] += vesting_balance_obj.balance.amount;
                 dlog("Vesting balance for account: ${owner}, amount: ${amount}",
@@ -1452,27 +1286,13 @@ void database::perform_chain_maintenance(const signed_block& next_block, const g
             const account_object& opinion_account = *opinion_account_ptr;
 
             const auto& stats = stake_account.statistics(d);
-            uint64_t voting_stake = 0;
+            uint64_t voting_stake = stats.total_core_in_orders.value
+                  + (stake_account.cashback_vb.valid() ? (*stake_account.cashback_vb)(d).balance.amount.value: 0)
+                  + d.get_balance(stake_account.get_id(), asset_id_type()).amount.value;
 
             auto itr = vesting_amounts.find(stake_account.id);
             if (itr != vesting_amounts.end())
                 voting_stake += itr->second.value;
-
-            if(d.head_block_time() >= HARDFORK_GPOS_TIME)
-            {
-               if (itr == vesting_amounts.end())
-                  return;
-
-               auto vesting_factor = d.calculate_vesting_factor(stake_account);
-               voting_stake = (uint64_t)floor(voting_stake * vesting_factor);
-            }
-            else
-            {
-               voting_stake += stats.total_core_in_orders.value
-                               + (stake_account.cashback_vb.valid() ? (*stake_account.cashback_vb)(d).balance.amount.value : 0)
-                               + d.get_balance(stake_account.get_id(), asset_id_type()).amount.value;
-            }
-
             for( vote_id_type id : opinion_account.options.votes )
             {
                uint32_t offset = id.instance();

libraries/chain/db_management.cpp

@@ -47,33 +47,86 @@ database::~database()
    clear_pending();
 }
 
-void database::reindex(fc::path data_dir, const genesis_state_type& initial_allocation)
+// Right now, we leave undo_db enabled when replaying when the bookie plugin is
-{ try {
+// enabled.  It depends on new/changed/removed object notifications, and those are
-   ilog( "reindexing blockchain" );
+// only fired when the undo_db is enabled.
-   wipe(data_dir, false);
+// So we use this helper object to disable undo_db only if it is not forbidden
-   open(data_dir, [&initial_allocation]{return initial_allocation;});
+// with _slow_replays flag.
+class auto_undo_enabler
+{
+    const bool _slow_replays;
+    undo_database& _undo_db;
+    bool _disabled;
+public:
+    auto_undo_enabler(bool slow_replays, undo_database& undo_db) :
+        _slow_replays(slow_replays),
+        _undo_db(undo_db),
+        _disabled(false)
+    {
+    }
 
-   auto start = fc::time_point::now();
+    ~auto_undo_enabler()
+    {
+        try{
+            enable();
+        } FC_CAPTURE_AND_LOG(("undo_db enabling crash"))
+    }
+
+    void enable()
+    {
+        if(!_disabled)
+            return;
+        _undo_db.enable();
+        _disabled = false;
+    }
+
+    void disable()
+    {
+        if(_disabled)
+            return;
+        if(_slow_replays)
+            return;
+        _undo_db.disable();
+        _disabled = true;
+    }
+};
+
+void database::reindex( fc::path data_dir )
+{ try {
    auto last_block = _block_id_to_block.last();
    if( !last_block ) {
       elog( "!no last block" );
       edump((last_block));
       return;
    }
+   if( last_block->block_num() <= head_block_num()) return;
 
+   ilog( "reindexing blockchain" );
+   auto start = fc::time_point::now();
    const auto last_block_num = last_block->block_num();
+   uint32_t flush_point = last_block_num < 10000 ? 0 : last_block_num - 10000;
+   uint32_t undo_point = last_block_num < 50 ? 0 : last_block_num - 50;
 
-   ilog( "Replaying blocks..." );
+   ilog( "Replaying blocks, starting at ${next}...", ("next",head_block_num() + 1) );
-   // Right now, we leave undo_db enabled when replaying when the bookie plugin is 
+   auto_undo_enabler undo(_slow_replays, _undo_db);
-   // enabled.  It depends on new/changed/removed object notifications, and those are 
+   if( head_block_num() >= undo_point )
-   // only fired when the undo_db is enabled
-   if (!_slow_replays)
-      _undo_db.disable();
-   for( uint32_t i = 1; i <= last_block_num; ++i )
    {
-      if( i == 1 || 
+      if( head_block_num() > 0 )
-          i % 10000 == 0 ) 
+         _fork_db.start_block( *fetch_block_by_number( head_block_num() ) );
-         std::cerr << "   " << double(i*100)/last_block_num << "%   "<< i << " of " <<last_block_num<<"   \n";
+   }
+   else
+   {
+       undo.disable();
+   }
+   for( uint32_t i = head_block_num() + 1; i <= last_block_num; ++i )
+   {
+      if( i % 10000 == 0 ) std::cerr << "   " << double(i*100)/last_block_num << "%   "<<i << " of " <<last_block_num<<"   \n";
+      if( i == flush_point )
+      {
+         ilog( "Writing database to disk at block ${i}", ("i",i) );
+         flush();
+         ilog( "Done" );
+      }
       fc::optional< signed_block > block = _block_id_to_block.fetch_by_number(i);
       if( !block.valid() )
       {
@@ -94,15 +147,8 @@ void database::reindex(fc::path data_dir, const genesis_state_type& initial_allo
          wlog( "Dropped ${n} blocks from after the gap", ("n", dropped_count) );
          break;
       }
-      if (_slow_replays)
+      if( i < undo_point && !_slow_replays)
-         push_block(*block, skip_fork_db |
+      {
-                            skip_witness_signature |
-                            skip_transaction_signatures |
-                            skip_transaction_dupe_check |
-                            skip_tapos_check |
-                            skip_witness_schedule_check |
-                            skip_authority_check);
-      else
          apply_block(*block, skip_witness_signature |
                              skip_transaction_signatures |
                              skip_transaction_dupe_check |
@@ -110,8 +156,18 @@ void database::reindex(fc::path data_dir, const genesis_state_type& initial_allo
                              skip_witness_schedule_check |
                              skip_authority_check);
       }
-   if (!_slow_replays)
+      else
-     _undo_db.enable();
+      {
+         undo.enable();
+         push_block(*block, skip_witness_signature |
+                            skip_transaction_signatures |
+                            skip_transaction_dupe_check |
+                            skip_tapos_check |
+                            skip_witness_schedule_check |
+                            skip_authority_check);
+      }
+   }
+   undo.enable();
    auto end = fc::time_point::now();
    ilog( "Done reindexing, elapsed time: ${t} sec", ("t",double((end-start).count())/1000000.0 ) );
 } FC_CAPTURE_AND_RETHROW( (data_dir) ) }
@@ -119,7 +175,9 @@ void database::reindex(fc::path data_dir, const genesis_state_type& initial_allo
 void database::wipe(const fc::path& data_dir, bool include_blocks)
 {
    ilog("Wiping database", ("include_blocks", include_blocks));
+   if (_opened) {
      close();
+   }
    object_database::wipe(data_dir);
    if( include_blocks )
       fc::remove_all( data_dir / "database" );
@@ -127,10 +185,29 @@ void database::wipe(const fc::path& data_dir, bool include_blocks)
 
 void database::open(
    const fc::path& data_dir,
-   std::function<genesis_state_type()> genesis_loader)
+   std::function<genesis_state_type()> genesis_loader,
+   const std::string& db_version)
 {
    try
    {
+      bool wipe_object_db = false;
+      if( !fc::exists( data_dir / "db_version" ) )
+         wipe_object_db = true;
+      else
+      {
+         std::string version_string;
+         fc::read_file_contents( data_dir / "db_version", version_string );
+         wipe_object_db = ( version_string != db_version );
+      }
+      if( wipe_object_db ) {
+          ilog("Wiping object_database due to missing or wrong version");
+          object_database::wipe( data_dir );
+          std::ofstream version_file( (data_dir / "db_version").generic_string().c_str(),
+                                      std::ios::out | std::ios::binary | std::ios::trunc );
+          version_file.write( db_version.c_str(), db_version.size() );
+          version_file.close();
+      }
+
       object_database::open(data_dir);
 
       _block_id_to_block.open(data_dir / "database" / "block_num_to_block");
@@ -138,24 +215,24 @@ void database::open(
       if( !find(global_property_id_type()) )
          init_genesis(genesis_loader());
 
-      fc::optional<signed_block> last_block = _block_id_to_block.last();
+      fc::optional<block_id_type> last_block = _block_id_to_block.last_id();
       if( last_block.valid() )
       {
-         _fork_db.start_block( *last_block );
+         FC_ASSERT( *last_block >= head_block_id(),
-         idump((last_block->id())(last_block->block_num()));
+                    "last block ID does not match current chain state",
-         idump((head_block_id())(head_block_num()));
+                    ("last_block->id", last_block)("head_block_id",head_block_num()) );
-         if( last_block->id() != head_block_id() )
+         reindex( data_dir );
-         {
-              FC_ASSERT( head_block_num() == 0, "last block ID does not match current chain state",
-                         ("last_block->id", last_block->id())("head_block_num",head_block_num()) );
-         }
       }
+      _opened = true;
    }
    FC_CAPTURE_LOG_AND_RETHROW( (data_dir) )
 }
 
 void database::close(bool rewind)
 {
+   if (!_opened)
+      return;
+      
    // TODO:  Save pending tx's on close()
    clear_pending();
 
@@ -169,17 +246,9 @@ void database::close(bool rewind)
 
          while( head_block_num() > cutoff )
          {
-         //   elog("pop");
             block_id_type popped_block_id = head_block_id();
             pop_block();
             _fork_db.remove(popped_block_id); // doesn't throw on missing
-            try
-            {
-               _block_id_to_block.remove(popped_block_id);
-            }
-            catch (const fc::key_not_found_exception&)
-            {
-            }
          }
       }
       catch ( const fc::exception& e )
@@ -200,6 +269,8 @@ void database::close(bool rewind)
       _block_id_to_block.close();
 
    _fork_db.reset();
+
+   _opened = false;
 }
 
 void database::force_slow_replays()
@@ -208,4 +279,31 @@ void database::force_slow_replays()
    _slow_replays = true;
 }
 
+void database::check_ending_lotteries()
+{
+   try {
+      const auto& lotteries_idx = get_index_type<asset_index>().indices().get<active_lotteries>();      
+      for( auto checking_asset: lotteries_idx )
+      {
+         FC_ASSERT( checking_asset.is_lottery() );
+         FC_ASSERT( checking_asset.lottery_options->is_active );
+         FC_ASSERT( checking_asset.lottery_options->end_date != time_point_sec() );
+         if( checking_asset.lottery_options->end_date > head_block_time() ) continue;
+         checking_asset.end_lottery(*this);
+      }
+   } catch( ... ) {}
+}
+
+void database::check_lottery_end_by_participants( asset_id_type asset_id )
+{
+   try {
+      asset_object asset_to_check = asset_id( *this );
+      auto asset_dyn_props = asset_to_check.dynamic_data( *this );
+      FC_ASSERT( asset_dyn_props.current_supply == asset_to_check.options.max_supply );
+      FC_ASSERT( asset_to_check.is_lottery() );
+      FC_ASSERT( asset_to_check.lottery_options->ending_on_soldout );
+      asset_to_check.end_lottery( *this );
+   } catch( ... ) {}
+}
+
 } }

libraries/chain/db_notify.cpp

@@ -269,6 +269,22 @@ struct get_impacted_account_visitor
       _impacted.insert( op.affiliate );
    }
    void operator()( const affiliate_referral_payout_operation& op ) { }
+   void operator()( const lottery_asset_create_operation& op ) {}
+   void operator()( const ticket_purchase_operation& op )
+   {
+      _impacted.insert( op.buyer );
+   }
+   void operator()( const lottery_reward_operation& op ) {
+      _impacted.insert( op.winner );
+   }
+   void operator()( const lottery_end_operation& op ) {
+      for( auto participant : op.participants ) {
+         _impacted.insert(participant.first);
+      }
+   }
+   void operator()( const sweeps_vesting_claim_operation& op ) {
+      _impacted.insert( op.account );
+   }
 };
 
 void operation_get_impacted_accounts( const operation& op, flat_set<account_id_type>& result )

libraries/chain/db_update.cpp

@@ -43,43 +43,13 @@
 
 namespace graphene { namespace chain {
 
-void database::update_global_dynamic_data( const signed_block& b )
+void database::update_global_dynamic_data( const signed_block& b, const uint32_t missed_blocks )
 {
    const dynamic_global_property_object& _dgp = dynamic_global_property_id_type(0)(*this);
    const global_property_object& gpo = get_global_properties();
 
-   uint32_t missed_blocks = get_slot_at_time( b.timestamp );
-
-//#define DIRTY_TRICK // problem with missed_blocks can occur when "maintenance_interval" set to few minutes
-#ifdef DIRTY_TRICK
-   if (missed_blocks != 0) {
-#else
-   assert( missed_blocks != 0 );
-#endif
-// bad if-condition, this code needs to execute for both shuffled and rng algorithms
-//     if (gpo.parameters.witness_schedule_algorithm == GRAPHENE_WITNESS_SHUFFLED_ALGORITHM)
-//     {
-       missed_blocks--;
-       for( uint32_t i = 0; i < missed_blocks; ++i ) {
-          const auto& witness_missed = get_scheduled_witness( i+1 )(*this);
-          if(  witness_missed.id != b.witness ) {
-             /*
-             const auto& witness_account = witness_missed.witness_account(*this);
-             if( (fc::time_point::now() - b.timestamp) < fc::seconds(30) )
-                wlog( "Witness ${name} missed block ${n} around ${t}", ("name",witness_account.name)("n",b.block_num())("t",b.timestamp) );
-                */
-
-             modify( witness_missed, [&]( witness_object& w ) {
-               w.total_missed++;
-             });
-          }
-       }
-//   }
-#ifdef DIRTY_TRICK
-   }
-#endif
    // dynamic global properties updating
-   modify( _dgp, [&]( dynamic_global_property_object& dgp ){
+   modify( _dgp, [&b,this,missed_blocks]( dynamic_global_property_object& dgp ){
       secret_hash_type::encoder enc;       
       fc::raw::pack( enc, dgp.random );       
       fc::raw::pack( enc, b.previous_secret );        
@@ -87,9 +57,10 @@ void database::update_global_dynamic_data( const signed_block& b )
 
       _random_number_generator = fc::hash_ctr_rng<secret_hash_type, 20>(dgp.random.data());
 
-      if( BOOST_UNLIKELY( b.block_num() == 1 ) )
+      const uint32_t block_num = b.block_num();
+      if( BOOST_UNLIKELY( block_num == 1 ) )
          dgp.recently_missed_count = 0;
-      else if( _checkpoints.size() && _checkpoints.rbegin()->first >= b.block_num() )
+      else if( _checkpoints.size() && _checkpoints.rbegin()->first >= block_num )
          dgp.recently_missed_count = 0;
       else if( missed_blocks )
          dgp.recently_missed_count += GRAPHENE_RECENTLY_MISSED_COUNT_INCREMENT*missed_blocks;
@@ -98,7 +69,7 @@ void database::update_global_dynamic_data( const signed_block& b )
       else if( dgp.recently_missed_count > 0 )
          dgp.recently_missed_count--;
 
-      dgp.head_block_number = b.block_num();
+      dgp.head_block_number = block_num;
       dgp.head_block_id = b.id();
       dgp.time = b.timestamp;
       dgp.current_witness = b.witness;

libraries/chain/db_witness_schedule.cpp

@@ -226,6 +226,22 @@ void database::update_witness_schedule(const signed_block& next_block)
       idump( ( double(total_time/1000000.0)/calls) );
 }
 
+uint32_t database::update_witness_missed_blocks( const signed_block& b )
+{
+   uint32_t missed_blocks = get_slot_at_time( b.timestamp );
+   FC_ASSERT( missed_blocks != 0, "Trying to push double-produced block onto current block?!" );
+   missed_blocks--;
+   const auto& witnesses = witness_schedule_id_type()(*this).current_shuffled_witnesses;
+   if( missed_blocks < witnesses.size() )
+      for( uint32_t i = 0; i < missed_blocks; ++i ) {
+         const auto& witness_missed = get_scheduled_witness( i+1 )(*this);
+         modify( witness_missed, []( witness_object& w ) {
+            w.total_missed++;
+         });
+      }
+   return missed_blocks;
+}
+
 uint32_t database::witness_participation_rate()const
 {
     const global_property_object& gpo = get_global_properties();

libraries/chain/event_object.cpp

@@ -553,30 +553,30 @@ namespace graphene { namespace chain {
 
 namespace fc { 
    // Manually reflect event_object to variant to properly reflect "state"
-   void to_variant(const graphene::chain::event_object& event_obj, fc::variant& v)
+   void to_variant(const graphene::chain::event_object& event_obj, fc::variant& v, uint32_t max_depth)
    {
       fc::mutable_variant_object o;
-      o("id", event_obj.id)
+      o("id", fc::variant(event_obj.id, max_depth))
-       ("name", event_obj.name)
+       ("name", fc::variant(event_obj.name, max_depth))
-       ("season", event_obj.season)
+       ("season", fc::variant(event_obj.season, max_depth))
-       ("start_time", event_obj.start_time)
+       ("start_time", fc::variant(event_obj.start_time, max_depth))
-       ("event_group_id", event_obj.event_group_id)
+       ("event_group_id", fc::variant(event_obj.event_group_id, max_depth))
-       ("scores", event_obj.scores)
+       ("scores", fc::variant(event_obj.scores, max_depth))
-       ("status", event_obj.get_status());
+       ("status", fc::variant(event_obj.get_status(), max_depth));
 
       v = o;
    }
 
    // Manually reflect event_object to variant to properly reflect "state"
-   void from_variant(const fc::variant& v, graphene::chain::event_object& event_obj)
+   void from_variant(const fc::variant& v, graphene::chain::event_object& event_obj, uint32_t max_depth)
    {
-      event_obj.id = v["id"].as<graphene::chain::event_id_type>();
+      event_obj.id = v["id"].as<graphene::chain::event_id_type>( max_depth );
-      event_obj.name = v["name"].as<graphene::chain::internationalized_string_type>();
+      event_obj.name = v["name"].as<graphene::chain::internationalized_string_type>( max_depth );
-      event_obj.season = v["season"].as<graphene::chain::internationalized_string_type>();
+      event_obj.season = v["season"].as<graphene::chain::internationalized_string_type>( max_depth );
-      event_obj.start_time = v["start_time"].as<optional<time_point_sec> >();
+      event_obj.start_time = v["start_time"].as<optional<time_point_sec> >( max_depth );
-      event_obj.event_group_id = v["event_group_id"].as<graphene::chain::event_group_id_type>();
+      event_obj.event_group_id = v["event_group_id"].as<graphene::chain::event_group_id_type>( max_depth );
-      event_obj.scores = v["scores"].as<std::vector<std::string>>();
+      event_obj.scores = v["scores"].as<std::vector<std::string>>( max_depth );
-      graphene::chain::event_status status = v["status"].as<graphene::chain::event_status>();
+      graphene::chain::event_status status = v["status"].as<graphene::chain::event_status>( max_depth );
       const_cast<int*>(event_obj.my->state_machine.current_state())[0] = (int)status;
    }
 } //end namespace fc

libraries/chain/game_object.cpp

@@ -549,33 +549,33 @@ namespace graphene { namespace chain {
 
 namespace fc { 
    // Manually reflect game_object to variant to properly reflect "state"
-   void to_variant(const graphene::chain::game_object& game_obj, fc::variant& v)
+   void to_variant(const graphene::chain::game_object& game_obj, fc::variant& v, uint32_t max_depth)
    {
       fc_elog(fc::logger::get("tournament"), "In game_obj to_variant");
       elog("In game_obj to_variant");
       fc::mutable_variant_object o;
-      o("id", game_obj.id)
+      o("id", fc::variant(game_obj.id,  max_depth ))
-       ("match_id", game_obj.match_id)
+       ("match_id", fc::variant(game_obj.match_id,  max_depth ))
-       ("players", game_obj.players)
+       ("players", fc::variant(game_obj.players,  max_depth ))
-       ("winners", game_obj.winners)
+       ("winners", fc::variant(game_obj.winners,  max_depth ))
-       ("game_details", game_obj.game_details)
+       ("game_details", fc::variant(game_obj.game_details,  max_depth ))
-       ("next_timeout", game_obj.next_timeout)
+       ("next_timeout", fc::variant(game_obj.next_timeout,  max_depth ))
-       ("state", game_obj.get_state());
+       ("state", fc::variant(game_obj.get_state(),  max_depth ));
 
       v = o;
    }
 
    // Manually reflect game_object to variant to properly reflect "state"
-   void from_variant(const fc::variant& v, graphene::chain::game_object& game_obj)
+   void from_variant(const fc::variant& v, graphene::chain::game_object& game_obj, uint32_t max_depth)
    {
       fc_elog(fc::logger::get("tournament"), "In game_obj from_variant");
-      game_obj.id = v["id"].as<graphene::chain::game_id_type>();
+      game_obj.id = v["id"].as<graphene::chain::game_id_type>(  max_depth  );
-      game_obj.match_id = v["match_id"].as<graphene::chain::match_id_type>();
+      game_obj.match_id = v["match_id"].as<graphene::chain::match_id_type>(  max_depth  );
-      game_obj.players = v["players"].as<std::vector<graphene::chain::account_id_type> >();
+      game_obj.players = v["players"].as<std::vector<graphene::chain::account_id_type> >(  max_depth  );
-      game_obj.winners = v["winners"].as<flat_set<graphene::chain::account_id_type> >();
+      game_obj.winners = v["winners"].as<flat_set<graphene::chain::account_id_type> >(  max_depth  );
-      game_obj.game_details = v["game_details"].as<graphene::chain::game_specific_details>();
+      game_obj.game_details = v["game_details"].as<graphene::chain::game_specific_details>(  max_depth  );
-      game_obj.next_timeout = v["next_timeout"].as<fc::optional<time_point_sec> >();
+      game_obj.next_timeout = v["next_timeout"].as<fc::optional<time_point_sec> >(  max_depth  );
-      graphene::chain::game_state state = v["state"].as<graphene::chain::game_state>();
+      graphene::chain::game_state state = v["state"].as<graphene::chain::game_state>(  max_depth  );
       const_cast<int*>(game_obj.my->state_machine.current_state())[0] = (int)state;
    }
 } //end namespace fc

libraries/chain/get_config.cpp

@@ -103,11 +103,11 @@ fc::variant_object get_config()
    result[ "GRAPHENE_DEFAULT_WITNESS_PAY_VESTING_SECONDS" ] = GRAPHENE_DEFAULT_WITNESS_PAY_VESTING_SECONDS;
    result[ "GRAPHENE_DEFAULT_WORKER_BUDGET_PER_DAY" ] = GRAPHENE_DEFAULT_WORKER_BUDGET_PER_DAY;
    result[ "GRAPHENE_MAX_INTEREST_APR" ] = GRAPHENE_MAX_INTEREST_APR;
-   result[ "GRAPHENE_COMMITTEE_ACCOUNT" ] = GRAPHENE_COMMITTEE_ACCOUNT;
+   result[ "GRAPHENE_COMMITTEE_ACCOUNT" ] = fc::variant(GRAPHENE_COMMITTEE_ACCOUNT, GRAPHENE_MAX_NESTED_OBJECTS);
-   result[ "GRAPHENE_WITNESS_ACCOUNT" ] = GRAPHENE_WITNESS_ACCOUNT;
+   result[ "GRAPHENE_WITNESS_ACCOUNT" ] = fc::variant(GRAPHENE_WITNESS_ACCOUNT, GRAPHENE_MAX_NESTED_OBJECTS);
-   result[ "GRAPHENE_RELAXED_COMMITTEE_ACCOUNT" ] = GRAPHENE_RELAXED_COMMITTEE_ACCOUNT;
+   result[ "GRAPHENE_RELAXED_COMMITTEE_ACCOUNT" ] = fc::variant(GRAPHENE_RELAXED_COMMITTEE_ACCOUNT, GRAPHENE_MAX_NESTED_OBJECTS);
-   result[ "GRAPHENE_NULL_ACCOUNT" ] = GRAPHENE_NULL_ACCOUNT;
+   result[ "GRAPHENE_NULL_ACCOUNT" ] = fc::variant(GRAPHENE_NULL_ACCOUNT, GRAPHENE_MAX_NESTED_OBJECTS);
-   result[ "GRAPHENE_TEMP_ACCOUNT" ] = GRAPHENE_TEMP_ACCOUNT;
+   result[ "GRAPHENE_TEMP_ACCOUNT" ] = fc::variant(GRAPHENE_TEMP_ACCOUNT, GRAPHENE_MAX_NESTED_OBJECTS);
 
    return result;
 }

libraries/chain/hardfork.d/CORE-429.hf

@@ -0,0 +1,4 @@
+// bitshares-core #429 rounding issue when creating assets
+#ifndef HARDFORK_CORE_429_TIME
+#define HARDFORK_CORE_429_TIME (fc::time_point_sec( 1566784800 ))
+#endif

libraries/chain/hardfork.d/GPOS.hf

@@ -1,4 +0,0 @@
-// GPOS HARDFORK Friday, March 15, 2019 11:57:28 PM
-#ifndef HARDFORK_GPOS_TIME
-#define HARDFORK_GPOS_TIME (fc::time_point_sec( 1552694248 ))
-#endif

libraries/chain/hardfork.d/SWEEPS.hf

@@ -0,0 +1,3 @@
+#ifndef HARDFORK_SWEEPS_TIME
+#define HARDFORK_SWEEPS_TIME (fc::time_point_sec( 1566784800 ))
+#endif

libraries/chain/include/graphene/chain/account_object.hpp

@@ -278,6 +278,25 @@ namespace graphene { namespace chain {
     */
    class account_member_index : public secondary_index
    {
+      /* std::less::operator() is less efficient so using key_compare here.
+       * Let assume that it has two keys key1 and key2 and we want to insert key3.
+       * the insert function needs to first call std::less::operator() with key1 and key3.
+       * Assume std::less::operator()(key1, key3) returns false.
+       * It has to call std::less::operator() again with the keys switched, 
+       * std::less::operator()(key3, key1), to decide whether key1 is equal to key3 or 
+       * key3 is greater than key1. There are two calls to std::less::operator() to make
+       * a decision if the first call returns false.
+       * std::map::insert and std::set used key_compare, 
+       * there would be sufficient information to make the right decision using just one call.
+       */
+      class key_compare {
+      public:
+         inline bool operator()( const public_key_type& a, const public_key_type& b )const
+         {
+            return a.key_data < b.key_data;
+         }
+      };
+
       public:
          virtual void object_inserted( const object& obj ) override;
          virtual void object_removed( const object& obj ) override;
@@ -287,18 +306,18 @@ namespace graphene { namespace chain {
 
          /** given an account or key, map it to the set of accounts that reference it in an active or owner authority */
          map< account_id_type, set<account_id_type> > account_to_account_memberships;
-         map< public_key_type, set<account_id_type> > account_to_key_memberships;
+         map< public_key_type, set<account_id_type>, key_compare > account_to_key_memberships;
          /** some accounts use address authorities in the genesis block */
          map< address, set<account_id_type> >         account_to_address_memberships;
 
 
       protected:
          set<account_id_type>  get_account_members( const account_object& a )const;
-         set<public_key_type>  get_key_members( const account_object& a )const;
+         set<public_key_type, key_compare>  get_key_members( const account_object& a )const;
          set<address>          get_address_members( const account_object& a )const;
 
          set<account_id_type>  before_account_members;
-         set<public_key_type>  before_key_members;
+         set<public_key_type, key_compare>  before_key_members;
          set<address>          before_address_members;
    };
 
@@ -344,7 +363,30 @@ namespace graphene { namespace chain {
    };
 
 
-   struct by_account_asset;
+   /**
+    *  @brief This secondary index will allow fast access to the balance objects
+    *         that belonging to an account.
+    */
+   class balances_by_account_index : public secondary_index
+   {
+      public:
+         virtual void object_inserted( const object& obj ) override;
+         virtual void object_removed( const object& obj ) override;
+         virtual void about_to_modify( const object& before ) override;
+         virtual void object_modified( const object& after  ) override;
+
+         const map< asset_id_type, const account_balance_object* >& get_account_balances( const account_id_type& acct )const;
+         const account_balance_object* get_account_balance( const account_id_type& acct, const asset_id_type& asset )const;
+
+      private:
+         static const uint8_t  bits;
+         static const uint64_t mask;
+
+         /** Maps each account to its balance objects */
+         vector< vector< map< asset_id_type, const account_balance_object* > > > balances;
+         std::stack< object_id_type > ids_being_modified;
+   };
+   
    struct by_asset_balance;
    /**
     * @ingroup object_index
@@ -353,13 +395,6 @@ namespace graphene { namespace chain {
       account_balance_object,
       indexed_by<
          ordered_unique< tag<by_id>, member< object, object_id_type, &object::id > >,
-         ordered_unique< tag<by_account_asset>,
-            composite_key<
-               account_balance_object,
-               member<account_balance_object, account_id_type, &account_balance_object::owner>,
-               member<account_balance_object, asset_id_type, &account_balance_object::asset_type>
-            >
-         >,
          ordered_unique< tag<by_asset_balance>,
             composite_key<
                account_balance_object,
@@ -399,6 +434,26 @@ namespace graphene { namespace chain {
     */
    typedef generic_index<account_object, account_multi_index_type> account_index;
 
+   struct by_owner;
+   struct by_maintenance_seq;
+
+   /**
+    * @ingroup object_index
+    */
+   typedef multi_index_container<
+      account_statistics_object,
+      indexed_by<
+         ordered_unique< tag<by_id>, member< object, object_id_type, &object::id > >,
+         ordered_unique< tag<by_owner>,
+                         member< account_statistics_object, account_id_type, &account_statistics_object::owner > >
+      >
+   > account_stats_multi_index_type;
+
+   /**
+    * @ingroup object_index
+    */
+   typedef generic_index<account_statistics_object, account_stats_multi_index_type> account_stats_index;
+
    struct by_dividend_payout_account{}; // use when calculating pending payouts
    struct by_dividend_account_payout{}; // use when doing actual payouts
    struct by_account_dividend_payout{}; // use in get_full_accounts()

libraries/chain/include/graphene/chain/asset_evaluator.hpp

@@ -44,6 +44,22 @@ namespace graphene { namespace chain {
          bool fee_is_odd;
    };
 
+   class lottery_asset_create_evaluator : public evaluator<lottery_asset_create_evaluator>
+   {
+      public:
+         typedef lottery_asset_create_operation operation_type;
+
+         void_result do_evaluate( const lottery_asset_create_operation& o );
+         object_id_type do_apply( const lottery_asset_create_operation& o );
+
+         /** override the default behavior defined by generic_evalautor which is to
+          * post the fee to fee_paying_account_stats.pending_fees
+          */
+         virtual void pay_fee() override;
+      private:
+         bool fee_is_odd;
+   };
+
    class asset_issue_evaluator : public evaluator<asset_issue_evaluator>
    {
       public:

libraries/chain/include/graphene/chain/asset_object.hpp

@@ -40,6 +40,7 @@
 namespace graphene { namespace chain {
    class account_object;
    class database;
+   class transaction_evaluation_state;
    using namespace graphene::db;
    
    /**
@@ -62,6 +63,7 @@ namespace graphene { namespace chain {
 
          /// The number of shares currently in existence
          share_type current_supply;
+         optional<share_type> sweeps_tickets_sold;
          share_type confidential_supply; ///< total asset held in confidential balances
          share_type accumulated_fees; ///< fees accumulate to be paid out over time
          share_type fee_pool;         ///< in core asset
@@ -87,6 +89,8 @@ namespace graphene { namespace chain {
 
          /// @return true if this is a market-issued asset; false otherwise.
          bool is_market_issued()const { return bitasset_data_id.valid(); }
+         /// @return true if this is lottery asset; false otherwise.
+         bool is_lottery()const { return lottery_options.valid(); }
          /// @return true if users may request force-settlement of this market-issued asset; false otherwise
          bool can_force_settle()const { return !(options.flags & disable_force_settle); }
          /// @return true if the issuer of this market-issued asset may globally settle the asset; false otherwise
@@ -115,6 +119,8 @@ namespace graphene { namespace chain {
          string amount_to_pretty_string(const asset &amount)const
          { FC_ASSERT(amount.asset_id == id); return amount_to_pretty_string(amount.amount); }
          
+         uint32_t get_issuer_num()const
+         { return issuer.instance.value; } 
          /// Ticker symbol for this asset, i.e. "USD"
          string symbol;
          /// Maximum number of digits after the decimal point (must be <= 12)
@@ -124,6 +130,14 @@ namespace graphene { namespace chain {
 
          asset_options options;
 
+         // Extra data associated with lottery options. This field is non-null if is_lottery() returns true
+         optional<lottery_asset_options> lottery_options;
+         time_point_sec get_lottery_expiration() const;
+         vector<account_id_type> get_holders( database& db ) const;
+         void distribute_benefactors_part( database& db );
+         map< account_id_type, vector< uint16_t > > distribute_winners_part( database& db );
+         void distribute_sweeps_holders_part( database& db );
+         void end_lottery( database& db );
          
          /// Current supply, fee pool, and collected fees are stored in a separate object as they change frequently.
          asset_dynamic_data_id_type  dynamic_asset_data_id;
@@ -216,8 +230,16 @@ namespace graphene { namespace chain {
          share_type settlement_fund;
          ///@}
 
+         /// The time when @ref current_feed would expire
          time_point_sec feed_expiration_time()const
-         { return current_feed_publication_time + options.feed_lifetime_sec; }
+         {
+            uint32_t current_feed_seconds = current_feed_publication_time.sec_since_epoch();
+            if( std::numeric_limits<uint32_t>::max() - current_feed_seconds <= options.feed_lifetime_sec )
+               return time_point_sec::maximum();
+            else
+               return current_feed_publication_time + options.feed_lifetime_sec;
+         }
+         
          bool feed_is_expired_before_hardfork_615(time_point_sec current_time)const
          { return feed_expiration_time() >= current_time; }
          bool feed_is_expired(time_point_sec current_time)const
@@ -238,15 +260,59 @@ namespace graphene { namespace chain {
    //typedef flat_index<asset_bitasset_data_object> asset_bitasset_data_index;
    typedef generic_index<asset_bitasset_data_object, asset_bitasset_data_object_multi_index_type> asset_bitasset_data_index;
 
+   // used to sort active_lotteries index
+   struct lottery_asset_comparer
+   {
+      bool operator()(const asset_object& lhs, const asset_object& rhs) const
+      {
+         if ( !lhs.is_lottery() ) return false;
+         if ( !lhs.lottery_options->is_active && !rhs.is_lottery()) return true; // not active lotteries first, just assets then
+         if ( !lhs.lottery_options->is_active ) return false;
+         if ( lhs.lottery_options->is_active && ( !rhs.is_lottery() || !rhs.lottery_options->is_active ) ) return true;
+         return lhs.get_lottery_expiration() > rhs.get_lottery_expiration();
+      }
+   };
+
    struct by_symbol;
    struct by_type;
    struct by_issuer;
+   struct active_lotteries;
+   struct by_lottery;
+   struct by_lottery_owner;
    typedef multi_index_container<
       asset_object,
       indexed_by<
          ordered_unique< tag<by_id>, member< object, object_id_type, &object::id > >,
          ordered_unique< tag<by_symbol>, member<asset_object, string, &asset_object::symbol> >,
          ordered_non_unique< tag<by_issuer>, member<asset_object, account_id_type, &asset_object::issuer > >,
+         ordered_non_unique< tag<active_lotteries>,
+            identity< asset_object >,
+            lottery_asset_comparer
+         >,
+         ordered_unique< tag<by_lottery>,
+            composite_key<
+               asset_object,
+               const_mem_fun<asset_object, bool, &asset_object::is_lottery>,
+               member<object, object_id_type, &object::id>
+            >,
+            composite_key_compare<
+               std::greater< bool >,
+               std::greater< object_id_type >
+            >
+         >,
+         ordered_unique< tag<by_lottery_owner>,
+            composite_key<
+               asset_object,
+               const_mem_fun<asset_object, bool, &asset_object::is_lottery>,
+               const_mem_fun<asset_object, uint32_t, &asset_object::get_issuer_num>,
+               member<object, object_id_type, &object::id>
+            >,
+            composite_key_compare<
+               std::greater< bool >,
+               std::greater< uint32_t >,
+               std::greater< object_id_type >
+            >
+         >,
          ordered_unique< tag<by_type>,
             composite_key< asset_object,
                 const_mem_fun<asset_object, bool, &asset_object::is_market_issued>,
@@ -257,6 +323,7 @@ namespace graphene { namespace chain {
    > asset_object_multi_index_type;
    typedef generic_index<asset_object, asset_object_multi_index_type> asset_index;
 
+
    /**
     *  @brief contains properties that only apply to dividend-paying assets
     *
@@ -335,10 +402,83 @@ namespace graphene { namespace chain {
    
    
       
+   /**
+    * @ingroup object
+    */
+   class lottery_balance_object : public abstract_object<lottery_balance_object>
+   {
+      public:
+         static const uint8_t space_id = implementation_ids;
+         static const uint8_t type_id  = impl_lottery_balance_object_type;
+      
+         asset_id_type  lottery_id;
+         asset          balance;
+      
+         asset get_balance()const { return balance; }
+         void  adjust_balance(const asset& delta);
+   };
+   
+   
+   struct by_owner;
+   
+   /**
+    * @ingroup object_index
+    */
+   typedef multi_index_container<
+      lottery_balance_object,
+      indexed_by<
+         ordered_unique< tag<by_id>, member< object, object_id_type, &object::id > >,
+         ordered_non_unique< tag<by_owner>,
+            member<lottery_balance_object, asset_id_type, &lottery_balance_object::lottery_id>
+         >
+      >
+   > lottery_balance_index_type;
+   
+   /**
+    * @ingroup object_index
+    */
+   typedef generic_index<lottery_balance_object, lottery_balance_index_type> lottery_balance_index;
+   
+   
+   class sweeps_vesting_balance_object : public abstract_object<sweeps_vesting_balance_object>
+   {
+      public:
+         static const uint8_t space_id = implementation_ids;
+         static const uint8_t type_id  = impl_sweeps_vesting_balance_object_type;
+
+
+         account_id_type   owner;
+         uint64_t          balance;
+         asset_id_type     asset_id;
+         time_point_sec    last_claim_date;
+         
+         uint64_t get_balance()const { return balance; }
+         void  adjust_balance(const asset& delta);
+         asset available_for_claim() const { return asset( balance / SWEEPS_VESTING_BALANCE_MULTIPLIER , asset_id ); }
+   };
+
+   /**
+    * @ingroup object_index
+    */
+   typedef multi_index_container<
+      sweeps_vesting_balance_object,
+      indexed_by<
+         ordered_unique< tag<by_id>, member< object, object_id_type, &object::id > >,
+         ordered_non_unique< tag<by_owner>,
+            member<sweeps_vesting_balance_object, account_id_type, &sweeps_vesting_balance_object::owner>
+         >
+      >
+   > sweeps_vesting_balance_index_type;
+
+   /**
+    * @ingroup object_index
+    */
+   typedef generic_index<sweeps_vesting_balance_object, sweeps_vesting_balance_index_type> sweeps_vesting_balance_index;
+
 } } // graphene::chain
 
 FC_REFLECT_DERIVED( graphene::chain::asset_dynamic_data_object, (graphene::db::object),
-                    (current_supply)(confidential_supply)(accumulated_fees)(fee_pool) )
+                    (current_supply)(sweeps_tickets_sold)(confidential_supply)(accumulated_fees)(fee_pool) )
 
 FC_REFLECT_DERIVED( graphene::chain::asset_bitasset_data_object, (graphene::db::object),
                     (feeds)
@@ -371,8 +511,15 @@ FC_REFLECT_DERIVED( graphene::chain::asset_object, (graphene::db::object),
                     (precision)
                     (issuer)
                     (options)
+                    (lottery_options)
                     (dynamic_asset_data_id)
                     (bitasset_data_id)
                     (buyback_account)
                     (dividend_data_id)
                   )
+
+FC_REFLECT_DERIVED( graphene::chain::lottery_balance_object, (graphene::db::object),
+                    (lottery_id)(balance) )
+
+FC_REFLECT_DERIVED( graphene::chain::sweeps_vesting_balance_object, (graphene::db::object),
+                    (owner)(balance)(asset_id)(last_claim_date) )

libraries/chain/include/graphene/chain/betting_market_object.hpp

@@ -36,10 +36,10 @@ namespace graphene { namespace chain {
 } }
 
 namespace fc { 
-   void to_variant(const graphene::chain::betting_market_object& betting_market_obj, fc::variant& v);
+   void to_variant(const graphene::chain::betting_market_object& betting_market_obj, fc::variant& v, uint32_t max_depth = 1);
-   void from_variant(const fc::variant& v, graphene::chain::betting_market_object& betting_market_obj);
+   void from_variant(const fc::variant& v, graphene::chain::betting_market_object& betting_market_obj, uint32_t max_depth = 1);
-   void to_variant(const graphene::chain::betting_market_group_object& betting_market_group_obj, fc::variant& v);
+   void to_variant(const graphene::chain::betting_market_group_object& betting_market_group_obj, fc::variant& v, uint32_t max_depth = 1);
-   void from_variant(const fc::variant& v, graphene::chain::betting_market_group_object& betting_market_group_obj);
+   void from_variant(const fc::variant& v, graphene::chain::betting_market_group_object& betting_market_group_obj, uint32_t max_depth = 1);
 } //end namespace fc
 
 namespace graphene { namespace chain {
@@ -109,8 +109,8 @@ class betting_market_group_object : public graphene::db::abstract_object< bettin
       template<typename Stream>
       friend Stream& operator>>( Stream& s, betting_market_group_object& betting_market_group_obj );
 
-      friend void ::fc::to_variant(const graphene::chain::betting_market_group_object& betting_market_group_obj, fc::variant& v);
+      friend void ::fc::to_variant(const graphene::chain::betting_market_group_object& betting_market_group_obj, fc::variant& v, uint32_t max_depth);
-      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::betting_market_group_object& betting_market_group_obj);
+      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::betting_market_group_object& betting_market_group_obj, uint32_t max_depth);
 
       void pack_impl(std::ostream& stream) const;
       void unpack_impl(std::istream& stream);
@@ -165,8 +165,8 @@ class betting_market_object : public graphene::db::abstract_object< betting_mark
       template<typename Stream>
       friend Stream& operator>>( Stream& s, betting_market_object& betting_market_obj );
 
-      friend void ::fc::to_variant(const graphene::chain::betting_market_object& betting_market_obj, fc::variant& v);
+      friend void ::fc::to_variant(const graphene::chain::betting_market_object& betting_market_obj, fc::variant& v, uint32_t max_depth);
-      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::betting_market_object& betting_market_obj);
+      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::betting_market_object& betting_market_obj, uint32_t max_depth);
 
       void pack_impl(std::ostream& stream) const;
       void unpack_impl(std::istream& stream);

libraries/chain/include/graphene/chain/block_database.hpp

@@ -26,6 +26,8 @@
 #include <graphene/chain/protocol/block.hpp>
 
 namespace graphene { namespace chain {
+   class index_entry;
+
    class block_database 
    {
       public:
@@ -44,6 +46,8 @@ namespace graphene { namespace chain {
          optional<signed_block> last()const;
          optional<block_id_type> last_id()const;
       private:
+         optional<index_entry> last_index_entry()const;
+         fc::path _index_filename;
          mutable std::fstream _blocks;
          mutable std::fstream _block_num_to_pos;
    };

libraries/chain/include/graphene/chain/config.hpp

@@ -151,7 +151,7 @@
 #define GRAPHENE_RECENTLY_MISSED_COUNT_INCREMENT             4
 #define GRAPHENE_RECENTLY_MISSED_COUNT_DECREMENT             3
 
-#define GRAPHENE_CURRENT_DB_VERSION                          "PPY2.2"
+#define GRAPHENE_CURRENT_DB_VERSION                          "PPY2.1"
 
 #define GRAPHENE_IRREVERSIBLE_THRESHOLD                      (70 * GRAPHENE_1_PERCENT)
 
@@ -211,6 +211,7 @@
                                                              { 10000000, 100000} } /* <= 1000: 10.00 */ 
 #define GRAPHENE_DEFAULT_BETTING_PERCENT_FEE (2 * GRAPHENE_1_PERCENT)
 #define GRAPHENE_DEFAULT_LIVE_BETTING_DELAY_TIME            5 // seconds
+#define GRAPHENE_MAX_NESTED_OBJECTS                         (200)
 #define TOURNAMENT_MIN_ROUND_DELAY                          0
 #define TOURNAMENT_MAX_ROUND_DELAY                          600
 #define TOURNAMENT_MIN_TIME_PER_COMMIT_MOVE                 0
@@ -226,6 +227,8 @@
 #define TOURNAMENT_MAX_WHITELIST_LENGTH                     1000
 #define TOURNAMENT_MAX_START_TIME_IN_FUTURE                 (60*60*24*7*4) // 1 month
 #define TOURNAMENT_MAX_START_DELAY                          (60*60*24*7) // 1 week
-#define GPOS_PERIOD                                         (60*60*24*30*6) // 6 months
+
-#define GPOS_SUBPERIOD                                      (60*60*24*30) // 1 month
+#define SWEEPS_DEFAULT_DISTRIBUTION_PERCENTAGE              (2*GRAPHENE_1_PERCENT)
-#define MIN_SON_MEMBER_COUNT                                15
+#define SWEEPS_DEFAULT_DISTRIBUTION_ASSET                   (graphene::chain::asset_id_type(0))
+#define SWEEPS_VESTING_BALANCE_MULTIPLIER                   100000000
+#define SWEEPS_ACCUMULATOR_ACCOUNT                          (graphene::chain::account_id_type(0))

libraries/chain/include/graphene/chain/database.hpp

@@ -91,10 +91,12 @@ namespace graphene { namespace chain {
           *
           * @param data_dir Path to open or create database in
           * @param genesis_loader A callable object which returns the genesis state to initialize new databases on
+          * @param db_version a version string that changes when the internal database format and/or logic is modified
           */
           void open(
              const fc::path& data_dir,
-             std::function<genesis_state_type()> genesis_loader );
+             std::function<genesis_state_type()> genesis_loader,
+             const std::string& db_version );
 
          /**
           * @brief Rebuild object graph from block history and open detabase
@@ -102,7 +104,7 @@ namespace graphene { namespace chain {
           * This method may be called after or instead of @ref database::open, and will rebuild the object graph by
           * replaying blockchain history. When this method exits successfully, the database will be open.
           */
-         void reindex(fc::path data_dir, const genesis_state_type& initial_allocation = genesis_state_type());
+         void reindex(fc::path data_dir);
 
          /**
           * @brief wipe Delete database from disk, and potentially the raw chain as well.
@@ -262,6 +264,9 @@ namespace graphene { namespace chain {
          void update_witness_schedule();
          void update_witness_schedule(const signed_block& next_block);
       
+         void check_lottery_end_by_participants( asset_id_type asset_id );
+         void check_ending_lotteries();
+
          //////////////////// db_getter.cpp ////////////////////
 
          const chain_id_type&                   get_chain_id()const;
@@ -271,7 +276,8 @@ namespace graphene { namespace chain {
          const dynamic_global_property_object&  get_dynamic_global_properties()const;
          const node_property_object&            get_node_properties()const;
          const fee_schedule&                    current_fee_schedule()const;
-
+         const std::vector<uint32_t>            get_winner_numbers( asset_id_type for_asset, uint32_t count_members, uint8_t count_winners ) const;
+         std::vector<uint32_t>                  get_seeds( asset_id_type for_asset, uint8_t count_winners )const;
          uint64_t                               get_random_bits( uint64_t bound );
 
          time_point_sec   head_block_time()const;
@@ -310,13 +316,26 @@ namespace graphene { namespace chain {
          asset get_balance(account_id_type owner, asset_id_type asset_id)const;
          /// This is an overloaded method.
          asset get_balance(const account_object& owner, const asset_object& asset_obj)const;
-
+         /**
+          * @brief Get balance connected with lottery asset; if assset isnt lottery - return asset(0, 0)
+          */
+         asset get_balance(asset_id_type lottery_id)const;
          /**
           * @brief Adjust a particular account's balance in a given asset by a delta
           * @param account ID of account whose balance should be adjusted
           * @param delta Asset ID and amount to adjust balance by
           */
          void adjust_balance(account_id_type account, asset delta);
+         /**
+          * @brief Adjust a lottery's balance in a given asset by a delta
+          * @param asset ID(should be lottery) balance should be adjusted
+          * @param delta Asset ID and amount to adjust balance by
+          */
+         void adjust_balance(asset_id_type lottery_id, asset delta);
+         /**
+          * @brief Adjust a particular account's sweeps vesting balance in a given asset by a delta
+          */
+         void adjust_sweeps_vesting_balance(account_id_type account, int64_t delta);
 
          /**
           * @brief Helper to make lazy deposit to CDD VBO.
@@ -471,8 +490,11 @@ namespace graphene { namespace chain {
          const witness_object& _validate_block_header( const signed_block& next_block )const;
          void create_block_summary(const signed_block& next_block);
 
+         //////////////////// db_witness_schedule.cpp ////////////////////
+         uint32_t update_witness_missed_blocks( const signed_block& b );
+
          //////////////////// db_update.cpp ////////////////////
-         void update_global_dynamic_data( const signed_block& b );
+         void update_global_dynamic_data( const signed_block& b, const uint32_t missed_blocks );
          void update_signing_witness(const witness_object& signing_witness, const signed_block& new_block);
          void update_last_irreversible_block();
          void clear_expired_transactions();
@@ -532,7 +554,7 @@ namespace graphene { namespace chain {
          uint32_t                          _current_block_num    = 0;
          uint16_t                          _current_trx_in_block = 0;
          uint16_t                          _current_op_in_trx    = 0;
-         uint16_t                          _current_virtual_op   = 0;
+         uint32_t                          _current_virtual_op   = 0;
 
          vector<uint64_t>                  _vote_tally_buffer;
          vector<uint64_t>                  _witness_count_histogram_buffer;
@@ -544,6 +566,15 @@ namespace graphene { namespace chain {
          node_property_object              _node_property_object;
          fc::hash_ctr_rng<secret_hash_type, 20> _random_number_generator;
          bool                              _slow_replays = false;
+
+         /**
+          * Whether database is successfully opened or not.
+          *
+          * The database is considered open when there's no exception
+          * or assertion fail during database::open() method, and
+          * database::close() has not been called, or failed during execution.
+          */
+         bool                              _opened = false;
    };
 
    namespace detail

libraries/chain/include/graphene/chain/event_group_object.hpp

@@ -26,7 +26,6 @@
 #include <graphene/chain/protocol/types.hpp>
 #include <graphene/db/object.hpp>
 #include <graphene/db/generic_index.hpp>
-
 #include <boost/multi_index/composite_key.hpp>
 
 namespace graphene { namespace chain {

libraries/chain/include/graphene/chain/event_object.hpp

@@ -35,8 +35,8 @@ namespace graphene { namespace chain {
 } }
 
 namespace fc { 
-   void to_variant(const graphene::chain::event_object& event_obj, fc::variant& v);
+   void to_variant(const graphene::chain::event_object& event_obj, fc::variant& v, uint32_t max_depth = 1);
-   void from_variant(const fc::variant& v, graphene::chain::event_object& event_obj);
+   void from_variant(const fc::variant& v, graphene::chain::event_object& event_obj, uint32_t max_depth = 1);
 } //end namespace fc
 
 namespace graphene { namespace chain {
@@ -76,8 +76,8 @@ class event_object : public graphene::db::abstract_object< event_object >
       template<typename Stream>
       friend Stream& operator>>( Stream& s, event_object& event_obj );
 
-      friend void ::fc::to_variant(const graphene::chain::event_object& event_obj, fc::variant& v);
+      friend void ::fc::to_variant(const graphene::chain::event_object& event_obj, fc::variant& v, uint32_t max_depth);
-      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::event_object& event_obj);
+      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::event_object& event_obj, uint32_t max_depth);
 
       void pack_impl(std::ostream& stream) const;
       void unpack_impl(std::istream& stream);

libraries/chain/include/graphene/chain/game_object.hpp

@@ -36,8 +36,8 @@ namespace graphene { namespace chain {
 } }
 
 namespace fc { 
-   void to_variant(const graphene::chain::game_object& game_obj, fc::variant& v);
+   void to_variant(const graphene::chain::game_object& game_obj, fc::variant& v, uint32_t max_depth = 1);
-   void from_variant(const fc::variant& v, graphene::chain::game_object& game_obj);
+   void from_variant(const fc::variant& v, graphene::chain::game_object& game_obj, uint32_t max_depth = 1);
 } //end namespace fc
 
 namespace graphene { namespace chain {
@@ -92,8 +92,8 @@ namespace graphene { namespace chain {
       template<typename Stream>
       friend Stream& operator>>( Stream& s, game_object& game_obj );
 
-      friend void ::fc::to_variant(const graphene::chain::game_object& game_obj, fc::variant& v);
+      friend void ::fc::to_variant(const graphene::chain::game_object& game_obj, fc::variant& v, uint32_t max_depth);
-      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::game_object& game_obj);
+      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::game_object& game_obj, uint32_t max_depth);
 
       void pack_impl(std::ostream& stream) const;
       void unpack_impl(std::istream& stream);

libraries/chain/include/graphene/chain/lottery_evaluator.hpp

@@ -0,0 +1,79 @@
+/*
+ * Copyright (c) 2017 Peerplays, Inc., and contributors.
+ *
+ * The MIT License
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#pragma once
+#include <graphene/chain/protocol/operations.hpp>
+#include <graphene/chain/evaluator.hpp>
+#include <graphene/chain/database.hpp>
+
+namespace graphene { namespace chain {
+
+   class ticket_purchase_evaluator : public evaluator<ticket_purchase_evaluator>
+   {
+      public:
+         typedef ticket_purchase_operation operation_type;
+
+         void_result do_evaluate( const ticket_purchase_operation& o );
+         void_result do_apply( const ticket_purchase_operation& o );
+
+         const asset_object* lottery;
+         const asset_dynamic_data_object* asset_dynamic_data;
+   };
+
+   class lottery_reward_evaluator : public evaluator<lottery_reward_evaluator>
+   {
+      public:
+         typedef lottery_reward_operation operation_type;
+
+         void_result do_evaluate( const lottery_reward_operation& o );
+         void_result do_apply( const lottery_reward_operation& o );
+
+         const asset_object* lottery;
+         const asset_dynamic_data_object* asset_dynamic_data;
+   };
+
+   class lottery_end_evaluator : public evaluator<lottery_end_evaluator>
+   {
+      public:
+         typedef lottery_end_operation operation_type;
+
+         void_result do_evaluate( const lottery_end_operation& o );
+         void_result do_apply( const lottery_end_operation& o );
+
+         const asset_object* lottery;
+         const asset_dynamic_data_object* asset_dynamic_data;
+   };
+   
+   class sweeps_vesting_claim_evaluator : public evaluator<sweeps_vesting_claim_evaluator>
+   {
+      public:
+         typedef sweeps_vesting_claim_operation operation_type;
+
+         void_result do_evaluate( const sweeps_vesting_claim_operation& o );
+         void_result do_apply( const sweeps_vesting_claim_operation& o );
+
+//         const asset_object* lottery;
+//         const asset_dynamic_data_object* asset_dynamic_data;
+   };
+
+} } // graphene::chain

libraries/chain/include/graphene/chain/match_object.hpp

@@ -12,8 +12,8 @@ namespace graphene { namespace chain {
 } }
 
 namespace fc { 
-   void to_variant(const graphene::chain::match_object& match_obj, fc::variant& v);
+   void to_variant(const graphene::chain::match_object& match_obj, fc::variant& v, uint32_t max_depth = 1);
-   void from_variant(const fc::variant& v, graphene::chain::match_object& match_obj);
+   void from_variant(const fc::variant& v, graphene::chain::match_object& match_obj, uint32_t max_depth = 1);
 } //end namespace fc
 
 namespace graphene { namespace chain {
@@ -84,8 +84,8 @@ namespace graphene { namespace chain {
       template<typename Stream>
       friend Stream& operator>>( Stream& s, match_object& match_obj );
 
-      friend void ::fc::to_variant(const graphene::chain::match_object& match_obj, fc::variant& v);
+      friend void ::fc::to_variant(const graphene::chain::match_object& match_obj, fc::variant& v, uint32_t max_depth);
-      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::match_object& match_obj);
+      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::match_object& match_obj, uint32_t max_depth);
 
       void pack_impl(std::ostream& stream) const;
       void unpack_impl(std::istream& stream);

libraries/chain/include/graphene/chain/operation_history_object.hpp

@@ -61,7 +61,7 @@ namespace graphene { namespace chain {
          /** the operation within the transaction */
          uint16_t          op_in_trx = 0;
          /** any virtual operations implied by operation in block */
-         uint16_t          virtual_op = 0;
+         uint32_t          virtual_op = 0;
    };
 
    /**

libraries/chain/include/graphene/chain/proposal_object.hpp

@@ -51,8 +51,9 @@ class proposal_object : public abstract_object<proposal_object>
       flat_set<account_id_type>     available_owner_approvals;
       flat_set<public_key_type>     available_key_approvals;
       account_id_type               proposer;
+      std::string                   fail_reason;
 
-      bool is_authorized_to_execute(database& db)const;
+      bool is_authorized_to_execute(database& db) const;
 };
 
 /**
@@ -94,4 +95,4 @@ typedef generic_index<proposal_object, proposal_multi_index_container> proposal_
 FC_REFLECT_DERIVED( graphene::chain::proposal_object, (graphene::chain::object),
                     (expiration_time)(review_period_time)(proposed_transaction)(required_active_approvals)
                     (available_active_approvals)(required_owner_approvals)(available_owner_approvals)
-                    (available_key_approvals)(proposer) )
+                    (available_key_approvals)(proposer)(fail_reason))

libraries/chain/include/graphene/chain/protocol/address.hpp

@@ -78,8 +78,8 @@ namespace graphene { namespace chain {
 
 namespace fc
 {
-   void to_variant( const graphene::chain::address& var,  fc::variant& vo );
+   void to_variant( const graphene::chain::address& var,  fc::variant& vo, uint32_t max_depth = 1 );
-   void from_variant( const fc::variant& var,  graphene::chain::address& vo );
+   void from_variant( const fc::variant& var,  graphene::chain::address& vo, uint32_t max_depth = 1 );
 }
 
 namespace std

libraries/chain/include/graphene/chain/protocol/asset_ops.hpp

@@ -26,9 +26,32 @@
 #include <graphene/chain/protocol/memo.hpp>
 
 namespace graphene { namespace chain { 
+   class database;
 
    bool is_valid_symbol( const string& symbol );
 
+   struct benefactor   {
+      account_id_type id;
+      uint16_t share; // percent * GRAPHENE_1_PERCENT
+      benefactor() = default;
+      benefactor( const benefactor & ) = default;
+      benefactor( account_id_type _id, uint16_t _share ) : id( _id ), share( _share ) {}
+   };
+
+   struct lottery_asset_options
+   {
+      std::vector<benefactor> benefactors;
+      asset_id_type           owner;
+      // specifying winning tickets as shares that will be issued
+      std::vector<uint16_t>   winning_tickets;
+      asset                   ticket_price;
+      time_point_sec          end_date;
+      bool                    ending_on_soldout;
+      bool                    is_active;
+
+      void validate()const;
+   };
+
    /**
     * @brief The asset_options struct contains options available on all assets in the network
     *
@@ -191,6 +214,7 @@ namespace graphene { namespace chain {
       optional<bitasset_options> bitasset_opts;
       /// For BitAssets, set this to true if the asset implements a @ref prediction_market; false otherwise
       bool is_prediction_market = false;
+      // containing lottery_asset_options now
       extensions_type extensions;
 
       account_id_type fee_payer()const { return issuer; }
@@ -198,6 +222,41 @@ namespace graphene { namespace chain {
       share_type      calculate_fee( const fee_parameters_type& k )const;
    };
 
+   ///Operation for creation of lottery
+   struct lottery_asset_create_operation : public base_operation
+   {
+      struct fee_parameters_type { 
+         uint64_t lottery_asset  = 20   * GRAPHENE_BLOCKCHAIN_PRECISION;
+         uint32_t price_per_kbyte = 10; /// only required for large lottery names.
+      };
+
+      asset                   fee;
+      /// This account must sign and pay the fee for this operation. Later, this account may update the asset
+      account_id_type         issuer;
+      /// The ticker symbol of this asset
+      string                  symbol;
+      /// Number of digits to the right of decimal point, must be less than or equal to 12
+      uint8_t                 precision = 0;
+
+      /// Options common to all assets.
+      ///
+      /// @note common_options.core_exchange_rate technically needs to store the asset ID of this new asset. Since this
+      /// ID is not known at the time this operation is created, create this price as though the new asset has instance
+      /// ID 1, and the chain will overwrite it with the new asset's ID.
+      asset_options              common_options;
+      /// Options only available for BitAssets. MUST be non-null if and only if the @ref market_issued flag is set in
+      /// common_options.flags
+      optional<bitasset_options> bitasset_opts;
+      /// For BitAssets, set this to true if the asset implements a @ref prediction_market; false otherwise
+      bool is_prediction_market = false;
+      // containing lottery_asset_options now
+      lottery_asset_options extensions;
+
+      account_id_type fee_payer()const { return issuer; }
+      void            validate()const;
+      share_type      calculate_fee( const fee_parameters_type& k )const;
+   };
+
    /**
     *  @brief allows global settling of bitassets (black swan or prediction markets)
     *
@@ -398,7 +457,7 @@ namespace graphene { namespace chain {
     * BitAssets have some options which are not relevant to other asset types. This operation is used to update those
     * options an an existing BitAsset.
     *
-    * @pre @ref issuer MUST be an existing account and MUST match asset_object::issuer on @ref asset_to_update
+    * @pre @ref issuer MUST be an existing aaccount and MUST match asset_object::issuer on @ref asset_to_update
     * @pre @ref asset_to_update MUST be a BitAsset, i.e. @ref asset_object::is_market_issued() returns true
     * @pre @ref fee MUST be nonnegative, and @ref issuer MUST have a sufficient balance to pay it
     * @pre @ref new_options SHALL be internally consistent, as verified by @ref validate()
@@ -571,9 +630,27 @@ namespace graphene { namespace chain {
       void            validate()const;
    };
    
+   struct sweeps_vesting_claim_operation : public base_operation
+   {
+      struct fee_parameters_type {
+         uint64_t fee = 20 * GRAPHENE_BLOCKCHAIN_PRECISION;
+      };
+      
+      asset           fee;
+      account_id_type account;
+      asset           amount_to_claim;
+      extensions_type extensions;
+      
+      
+      account_id_type fee_payer()const { return account; }
+      void            validate()const {};
+   };
 
 } } // graphene::chain
 
+FC_REFLECT( graphene::chain::sweeps_vesting_claim_operation, (fee)(account)(amount_to_claim)(extensions) )
+FC_REFLECT( graphene::chain::sweeps_vesting_claim_operation::fee_parameters_type, (fee) )
+
 FC_REFLECT( graphene::chain::asset_claim_fees_operation, (fee)(issuer)(amount_to_claim)(extensions) )
 FC_REFLECT( graphene::chain::asset_claim_fees_operation::fee_parameters_type, (fee) )
 
@@ -610,8 +687,13 @@ FC_REFLECT( graphene::chain::bitasset_options,
             (extensions)
           )
 
+FC_REFLECT( graphene::chain::benefactor, (id)(share) )
+
+FC_REFLECT( graphene::chain::lottery_asset_options, (benefactors)(owner)(winning_tickets)(ticket_price)(end_date)(ending_on_soldout)(is_active) )
+
 
 FC_REFLECT( graphene::chain::asset_create_operation::fee_parameters_type, (symbol3)(symbol4)(long_symbol)(price_per_kbyte) )
+FC_REFLECT( graphene::chain::lottery_asset_create_operation::fee_parameters_type, (lottery_asset)(price_per_kbyte) )
 FC_REFLECT( graphene::chain::asset_global_settle_operation::fee_parameters_type, (fee) )
 FC_REFLECT( graphene::chain::asset_settle_operation::fee_parameters_type, (fee) )
 FC_REFLECT( graphene::chain::asset_settle_cancel_operation::fee_parameters_type, )
@@ -635,6 +717,16 @@ FC_REFLECT( graphene::chain::asset_create_operation,
             (is_prediction_market)
             (extensions)
           )
+FC_REFLECT( graphene::chain::lottery_asset_create_operation,
+            (fee)
+            (issuer)
+            (symbol)
+            (precision)
+            (common_options)
+            (bitasset_opts)
+            (is_prediction_market)
+            (extensions)
+          )
 FC_REFLECT( graphene::chain::asset_update_operation,
             (fee)
             (issuer)

libraries/chain/include/graphene/chain/protocol/chain_parameters.hpp

@@ -27,8 +27,6 @@
 #include <graphene/chain/protocol/types.hpp>
 #include <fc/smart_ref_fwd.hpp>
 
-#include <graphene/chain/hardfork.hpp>
-
 namespace graphene { namespace chain { struct fee_schedule; } }
 
 namespace graphene { namespace chain {
@@ -39,11 +37,9 @@ namespace graphene { namespace chain {
       optional< uint16_t >            betting_rake_fee_percentage;
       optional< flat_map<bet_multiplier_type, bet_multiplier_type> > permitted_betting_odds_increments;
       optional< uint16_t >            live_betting_delay_time;
-      /* gpos parameters */
+      optional< uint16_t >            sweeps_distribution_percentage;
-      optional < uint32_t >           gpos_period;
+      optional< asset_id_type >       sweeps_distribution_asset;
-      optional < uint32_t >           gpos_subperiod;
+      optional< account_id_type >     sweeps_vesting_accumulator_account;
-      optional < uint32_t >           gpos_period_start;
-      optional < uint16_t >           son_count;
    };
 
    struct chain_parameters
@@ -93,6 +89,7 @@ namespace graphene { namespace chain {
       uint32_t                maximum_tournament_start_time_in_future = TOURNAMENT_MAX_START_TIME_IN_FUTURE;
       uint32_t                maximum_tournament_start_delay      = TOURNAMENT_MAX_START_DELAY;
       uint16_t                maximum_tournament_number_of_wins   = TOURNAMENT_MAX_NUMBER_OF_WINS;
+      //
       extension<parameter_extension> extensions;
 
       /** defined in fee_schedule.cpp */
@@ -113,17 +110,14 @@ namespace graphene { namespace chain {
       inline uint16_t live_betting_delay_time()const {
          return extensions.value.live_betting_delay_time.valid() ? *extensions.value.live_betting_delay_time : GRAPHENE_DEFAULT_LIVE_BETTING_DELAY_TIME;
       }
-      inline uint32_t gpos_period()const {
+      inline uint16_t sweeps_distribution_percentage()const {
-         return extensions.value.gpos_period.valid() ? *extensions.value.gpos_period : GPOS_PERIOD; /// total seconds of current gpos period
+         return extensions.value.sweeps_distribution_percentage.valid() ? *extensions.value.sweeps_distribution_percentage : SWEEPS_DEFAULT_DISTRIBUTION_PERCENTAGE;
       }
-      inline uint32_t gpos_subperiod()const {
+      inline asset_id_type sweeps_distribution_asset()const {
-         return extensions.value.gpos_subperiod.valid() ? *extensions.value.gpos_subperiod : GPOS_SUBPERIOD; /// gpos_period % gpos_subperiod = 0
+         return extensions.value.sweeps_distribution_asset.valid() ? *extensions.value.sweeps_distribution_asset : SWEEPS_DEFAULT_DISTRIBUTION_ASSET;
       }
-      inline uint32_t gpos_period_start()const {
+      inline account_id_type sweeps_vesting_accumulator_account()const {
-         return extensions.value.gpos_period_start.valid() ? *extensions.value.gpos_period_start : HARDFORK_GPOS_TIME.sec_since_epoch(); /// current period start date
+         return extensions.value.sweeps_vesting_accumulator_account.valid() ? *extensions.value.sweeps_vesting_accumulator_account : SWEEPS_ACCUMULATOR_ACCOUNT;
-      }
-      inline uint16_t son_count()const {
-         return extensions.value.son_count.valid() ? *extensions.value.son_count : MIN_SON_MEMBER_COUNT;
       }
    };
 
@@ -135,10 +129,9 @@ FC_REFLECT( graphene::chain::parameter_extension,
    (betting_rake_fee_percentage)
    (permitted_betting_odds_increments)
    (live_betting_delay_time)
-   (gpos_period)
+   (sweeps_distribution_percentage)
-   (gpos_subperiod)
+   (sweeps_distribution_asset)
-   (gpos_period_start)
+   (sweeps_vesting_accumulator_account)
-   (son_count)
 )
 
 FC_REFLECT( graphene::chain::chain_parameters,

libraries/chain/include/graphene/chain/protocol/ext.hpp

@@ -145,9 +145,10 @@ namespace fc {
 template< typename T >
 struct graphene_extension_from_variant_visitor
 {
-   graphene_extension_from_variant_visitor( const variant_object& v, T& val )
+   graphene_extension_from_variant_visitor( const variant_object& v, T& val, uint32_t max_depth )
-      : vo( v ), value( val )
+      : vo( v ), value( val ), _max_depth(max_depth - 1)
    {
+      FC_ASSERT( max_depth > 0, "Recursion depth exceeded!" );
       count_left = vo.size();
    }
 
@@ -157,7 +158,7 @@ struct graphene_extension_from_variant_visitor
       auto it = vo.find(name);
       if( it != vo.end() )
       {
-         from_variant( it->value(), (value.*member) );
+         from_variant( it->value(), (value.*member), _max_depth );
          assert( count_left > 0 );    // x.find(k) returns true for n distinct values of k only if x.size() >= n
          --count_left;
       }
@@ -165,11 +166,12 @@ struct graphene_extension_from_variant_visitor
 
    const variant_object& vo;
    T& value;
+   const uint32_t _max_depth;
    mutable uint32_t count_left = 0;
 };
 
 template< typename T >
-void from_variant( const fc::variant& var, graphene::chain::extension<T>& value )
+void from_variant( const fc::variant& var, graphene::chain::extension<T>& value, uint32_t max_depth )
 {
    value = graphene::chain::extension<T>();
    if( var.is_null() )
@@ -180,7 +182,7 @@ void from_variant( const fc::variant& var, graphene::chain::extension<T>& value
       return;
    }
 
-   graphene_extension_from_variant_visitor<T> vtor( var.get_object(), value.value );
+   graphene_extension_from_variant_visitor<T> vtor( var.get_object(), value.value, max_depth );
    fc::reflector<T>::visit( vtor );
    FC_ASSERT( vtor.count_left == 0 );    // unrecognized extension throws here
 }
@@ -188,23 +190,23 @@ void from_variant( const fc::variant& var, graphene::chain::extension<T>& value
 template< typename T >
 struct graphene_extension_to_variant_visitor
 {
-   graphene_extension_to_variant_visitor( const T& v ) : value(v) {}
+   graphene_extension_to_variant_visitor( const T& v, uint32_t max_depth ) : value(v), mvo(max_depth) {}
 
    template<typename Member, class Class, Member (Class::*member)>
    void operator()( const char* name )const
    {
       if( (value.*member).valid() )
-         mvo[ name ] = (value.*member);
+         mvo( name, value.*member );
    }
 
    const T& value;
-   mutable mutable_variant_object mvo;
+   mutable limited_mutable_variant_object mvo;
 };
 
 template< typename T >
-void to_variant( const graphene::chain::extension<T>& value, fc::variant& var )
+void to_variant( const graphene::chain::extension<T>& value, fc::variant& var, uint32_t max_depth )
 {
-   graphene_extension_to_variant_visitor<T> vtor( value.value );
+   graphene_extension_to_variant_visitor<T> vtor( value.value, max_depth );
    fc::reflector<T>::visit( vtor );
    var = vtor.mvo;
 }

libraries/chain/include/graphene/chain/protocol/lottery_ops.hpp

@@ -0,0 +1,136 @@
+/*
+ * Copyright (c) 2017 Peerplays, Inc., and contributors.
+ *
+ * The MIT License
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#pragma once
+#include <graphene/chain/protocol/base.hpp>
+#include <graphene/chain/asset_object.hpp>
+
+namespace graphene { namespace chain {
+
+   /**
+    * @ingroup operations
+    */
+   struct ticket_purchase_operation : public base_operation
+   {
+      struct fee_parameters_type { 
+         uint64_t fee = 0;
+      };
+
+      asset                   fee;
+      // from what lottery is ticket
+      asset_id_type           lottery;
+      account_id_type         buyer;
+      // count of tickets to buy
+      uint64_t                tickets_to_buy;
+      // amount that can spent
+      asset                   amount;
+      
+      extensions_type         extensions;
+
+      account_id_type fee_payer()const { return buyer; }
+      void            validate()const;
+      share_type      calculate_fee( const fee_parameters_type& k )const;
+   };
+
+   /**
+    * @ingroup operations
+    */
+   struct lottery_reward_operation : public base_operation
+   {
+      struct fee_parameters_type { 
+         uint64_t fee = 0;
+      };
+
+      asset                   fee;
+      // from what lottery is ticket
+      asset_id_type           lottery;
+      // winner account
+      account_id_type         winner;
+      // amount that won 
+      asset                   amount;
+      // percentage of jackpot that user won
+      uint16_t                win_percentage;
+      // true if recieved from benefators section of lottery; false otherwise
+      bool                    is_benefactor_reward;
+
+      extensions_type         extensions;
+
+      account_id_type fee_payer()const { return account_id_type(); }
+      void            validate()const {};
+      share_type      calculate_fee( const fee_parameters_type& k )const { return k.fee; };
+   };
+
+   /**
+    * @ingroup operations
+    */
+   struct lottery_end_operation : public base_operation
+   {
+      struct fee_parameters_type {
+         uint64_t fee = 0;
+      };
+
+      asset                   fee;
+      // from what lottery is ticket
+      asset_id_type           lottery;
+
+      map<account_id_type, vector< uint16_t> > participants;
+      
+      extensions_type               extensions;
+
+      account_id_type fee_payer()const { return account_id_type(); }
+      void            validate() const {}
+      share_type      calculate_fee( const fee_parameters_type& k )const { return k.fee; } 
+   };
+
+} } // graphene::chain
+
+FC_REFLECT( graphene::chain::ticket_purchase_operation,
+            (fee)
+            (lottery)
+            (buyer)
+            (tickets_to_buy)
+            (amount)
+            (extensions)
+          )
+FC_REFLECT( graphene::chain::ticket_purchase_operation::fee_parameters_type, (fee) )
+
+
+FC_REFLECT( graphene::chain::lottery_reward_operation,
+            (fee)
+            (lottery)
+            (winner)
+            (amount)
+            (win_percentage)
+            (is_benefactor_reward)
+            (extensions)
+          )
+FC_REFLECT( graphene::chain::lottery_reward_operation::fee_parameters_type, (fee) )
+
+
+FC_REFLECT( graphene::chain::lottery_end_operation,
+            (fee)
+            (lottery)
+            (participants)
+            (extensions)
+          )
+FC_REFLECT( graphene::chain::lottery_end_operation::fee_parameters_type, (fee) )

libraries/chain/include/graphene/chain/protocol/operations.hpp

@@ -27,6 +27,7 @@
 #include <graphene/chain/protocol/affiliate.hpp>
 #include <graphene/chain/protocol/assert.hpp>
 #include <graphene/chain/protocol/asset_ops.hpp>
+#include <graphene/chain/protocol/lottery_ops.hpp>
 #include <graphene/chain/protocol/balance.hpp>
 #include <graphene/chain/protocol/custom.hpp>
 #include <graphene/chain/protocol/committee_member.hpp>
@@ -129,7 +130,12 @@ namespace graphene { namespace chain {
             sport_delete_operation,
             event_group_delete_operation,
             affiliate_payout_operation, // VIRTUAL
-            affiliate_referral_payout_operation // VIRTUAL
+            affiliate_referral_payout_operation, // VIRTUAL
+            lottery_asset_create_operation,
+            ticket_purchase_operation,
+            lottery_reward_operation,
+            lottery_end_operation,
+            sweeps_vesting_claim_operation
          > operation;
 
    /// @} // operations group

libraries/chain/include/graphene/chain/protocol/transaction.hpp

@@ -165,12 +165,30 @@ namespace graphene { namespace chain {
          uint32_t max_recursion = GRAPHENE_MAX_SIG_CHECK_DEPTH
          ) const;
 
-      flat_set<public_key_type> get_signature_keys( const chain_id_type& chain_id )const;
+      /**
+       * @brief Extract public keys from signatures with given chain ID.
+       * @param chain_id A chain ID
+       * @return Public keys
+       * @note If @ref signees is empty, E.G. when it's the first time calling
+       *       this function for the signed transaction, public keys will be
+       *       extracted with given chain ID, and be stored into the mutable
+       *       @ref signees field, then @ref signees will be returned;
+       *       otherwise, the @ref chain_id parameter will be ignored, and
+       *       @ref signees will be returned directly.
+       */
+      const flat_set<public_key_type>& get_signature_keys( const chain_id_type& chain_id )const;
 
+      /** Signatures */
       vector<signature_type> signatures;
 
-      /// Removes all operations and signatures
+      /** Public keys extracted from signatures */
-      void clear() { operations.clear(); signatures.clear(); }
+      mutable flat_set<public_key_type> signees;
+
+      /// Removes all operations, signatures and signees
+      void clear() { operations.clear(); signatures.clear(); signees.clear(); }
+
+      /// Removes all signatures and signees
+      void clear_signatures() { signatures.clear(); signees.clear(); }
    };
 
    void verify_authority( const vector<operation>& ops, const flat_set<public_key_type>& sigs,
@@ -209,5 +227,6 @@ namespace graphene { namespace chain {
 } } // graphene::chain
 
 FC_REFLECT( graphene::chain::transaction, (ref_block_num)(ref_block_prefix)(expiration)(operations)(extensions) )
+// Note: not reflecting signees field for backward compatibility; in addition, it should not be in p2p messages
 FC_REFLECT_DERIVED( graphene::chain::signed_transaction, (graphene::chain::transaction), (signatures) )
 FC_REFLECT_DERIVED( graphene::chain::processed_transaction, (graphene::chain::signed_transaction), (operation_results) )

libraries/chain/include/graphene/chain/protocol/types.hpp

@@ -171,7 +171,9 @@ namespace graphene { namespace chain {
       impl_pending_dividend_payout_balance_for_holder_object_type,
       impl_distributed_dividend_balance_data_type,
       impl_betting_market_position_object_type,
-      impl_global_betting_statistics_object_type
+      impl_global_betting_statistics_object_type,
+      impl_lottery_balance_object_type,
+      impl_sweeps_vesting_balance_object_type
    };
 
    //typedef fc::unsigned_int            object_id_type;
@@ -249,15 +251,19 @@ namespace graphene { namespace chain {
    class pending_dividend_payout_balance_for_holder_object;
    class betting_market_position_object;
    class global_betting_statistics_object;
+   class lottery_balance_object;
+   class sweeps_vesting_balance_object;
 
    typedef object_id< implementation_ids, impl_global_property_object_type,            global_property_object>          global_property_id_type;
    typedef object_id< implementation_ids, impl_dynamic_global_property_object_type,    dynamic_global_property_object>  dynamic_global_property_id_type;
    typedef object_id< implementation_ids, impl_asset_dynamic_data_type,                asset_dynamic_data_object>       asset_dynamic_data_id_type;
    typedef object_id< implementation_ids, impl_asset_bitasset_data_type,               asset_bitasset_data_object>      asset_bitasset_data_id_type;
    typedef object_id< implementation_ids, impl_asset_dividend_data_type,               asset_dividend_data_object>      asset_dividend_data_id_type;
-   typedef object_id< implementation_ids, impl_pending_dividend_payout_balance_for_holder_object_type,     pending_dividend_payout_balance_for_holder_object>                pending_dividend_payout_balance_for_holder_object_type;
+   typedef object_id< implementation_ids,
+                      impl_pending_dividend_payout_balance_for_holder_object_type,
+                      pending_dividend_payout_balance_for_holder_object>                                                pending_dividend_payout_balance_for_holder_object_type;
    typedef object_id< implementation_ids, impl_account_balance_object_type,            account_balance_object>          account_balance_id_type;
-   typedef object_id< implementation_ids, impl_account_statistics_object_type,account_statistics_object>                account_statistics_id_type;
+   typedef object_id< implementation_ids, impl_account_statistics_object_type,         account_statistics_object>       account_statistics_id_type;
    typedef object_id< implementation_ids, impl_transaction_object_type,                transaction_object>              transaction_obj_id_type;
    typedef object_id< implementation_ids, impl_block_summary_object_type,              block_summary_object>            block_summary_id_type;
 
@@ -273,6 +279,8 @@ namespace graphene { namespace chain {
    typedef object_id< implementation_ids, impl_fba_accumulator_object_type, fba_accumulator_object >                    fba_accumulator_id_type;
    typedef object_id< implementation_ids, impl_betting_market_position_object_type, betting_market_position_object >    betting_market_position_id_type;
    typedef object_id< implementation_ids, impl_global_betting_statistics_object_type, global_betting_statistics_object > global_betting_statistics_id_type;
+   typedef object_id< implementation_ids, impl_lottery_balance_object_type, lottery_balance_object >                    lottery_balance_id_type;
+   typedef object_id< implementation_ids, impl_sweeps_vesting_balance_object_type, sweeps_vesting_balance_object>       sweeps_vesting_balance_id_type;
 
    typedef fc::array<char, GRAPHENE_MAX_ASSET_SYMBOL_LENGTH>    symbol_type;
    typedef fc::ripemd160                                        block_id_type;
@@ -359,12 +367,12 @@ namespace graphene { namespace chain {
 
 namespace fc
 {
-    void to_variant( const graphene::chain::public_key_type& var,  fc::variant& vo );
+    void to_variant( const graphene::chain::public_key_type& var,  fc::variant& vo, uint32_t max_depth = 2 );
-    void from_variant( const fc::variant& var,  graphene::chain::public_key_type& vo );
+    void from_variant( const fc::variant& var,  graphene::chain::public_key_type& vo, uint32_t max_depth = 2 );
-    void to_variant( const graphene::chain::extended_public_key_type& var, fc::variant& vo );
+    void to_variant( const graphene::chain::extended_public_key_type& var, fc::variant& vo, uint32_t max_depth = 2 );
-    void from_variant( const fc::variant& var, graphene::chain::extended_public_key_type& vo );
+    void from_variant( const fc::variant& var, graphene::chain::extended_public_key_type& vo, uint32_t max_depth = 2 );
-    void to_variant( const graphene::chain::extended_private_key_type& var, fc::variant& vo );
+    void to_variant( const graphene::chain::extended_private_key_type& var, fc::variant& vo, uint32_t max_depth = 2 );
-    void from_variant( const fc::variant& var, graphene::chain::extended_private_key_type& vo );
+    void from_variant( const fc::variant& var, graphene::chain::extended_private_key_type& vo, uint32_t max_depth = 2 );
 }
 
 FC_REFLECT( graphene::chain::public_key_type, (key_data) )
@@ -427,6 +435,8 @@ FC_REFLECT_ENUM( graphene::chain::impl_object_type,
                  (impl_distributed_dividend_balance_data_type)
                  (impl_betting_market_position_object_type)
                  (impl_global_betting_statistics_object_type)
+                 (impl_lottery_balance_object_type)
+                 (impl_sweeps_vesting_balance_object_type)
                )
 
 FC_REFLECT_TYPENAME( graphene::chain::share_type )

libraries/chain/include/graphene/chain/protocol/vesting.hpp

@@ -26,8 +26,6 @@
 
 namespace graphene { namespace chain { 
 
-   enum class vesting_balance_type { unspecified, gpos };
-
    struct linear_vesting_policy_initializer
    {
       /** while vesting begins on begin_timestamp, none may be claimed before vesting_cliff_seconds have passed */
@@ -74,7 +72,6 @@ namespace graphene { namespace chain {
       account_id_type             owner; ///< Who is able to withdraw the balance
       asset                       amount;
       vesting_policy_initializer  policy;
-      vesting_balance_type        balance_type;
 
       account_id_type   fee_payer()const { return creator; }
       void              validate()const
@@ -115,11 +112,9 @@ namespace graphene { namespace chain {
 FC_REFLECT( graphene::chain::vesting_balance_create_operation::fee_parameters_type, (fee) )
 FC_REFLECT( graphene::chain::vesting_balance_withdraw_operation::fee_parameters_type, (fee) )
 
-FC_REFLECT( graphene::chain::vesting_balance_create_operation, (fee)(creator)(owner)(amount)(policy)(balance_type) )
+FC_REFLECT( graphene::chain::vesting_balance_create_operation, (fee)(creator)(owner)(amount)(policy) )
 FC_REFLECT( graphene::chain::vesting_balance_withdraw_operation, (fee)(vesting_balance)(owner)(amount) )
 
 FC_REFLECT(graphene::chain::linear_vesting_policy_initializer, (begin_timestamp)(vesting_cliff_seconds)(vesting_duration_seconds) )
 FC_REFLECT(graphene::chain::cdd_vesting_policy_initializer, (start_claim)(vesting_seconds) )
 FC_REFLECT_TYPENAME( graphene::chain::vesting_policy_initializer )
-
-FC_REFLECT_ENUM( graphene::chain::vesting_balance_type, (unspecified)(gpos) )

libraries/chain/include/graphene/chain/protocol/vote.hpp

@@ -141,8 +141,8 @@ namespace fc
 
 class variant;
 
-void to_variant( const graphene::chain::vote_id_type& var, fc::variant& vo );
+void to_variant( const graphene::chain::vote_id_type& var, fc::variant& vo, uint32_t max_depth = 1 );
-void from_variant( const fc::variant& var, graphene::chain::vote_id_type& vo );
+void from_variant( const fc::variant& var, graphene::chain::vote_id_type& vo, uint32_t max_depth = 1 );
 
 } // fc
 

libraries/chain/include/graphene/chain/pts_address.hpp

@@ -73,6 +73,6 @@ FC_REFLECT( graphene::chain::pts_address, (addr) )
 
 namespace fc 
 { 
-   void to_variant( const graphene::chain::pts_address& var,  fc::variant& vo );
+   void to_variant( const graphene::chain::pts_address& var,  fc::variant& vo, uint32_t max_depth = 1 );
-   void from_variant( const fc::variant& var,  graphene::chain::pts_address& vo );
+   void from_variant( const fc::variant& var,  graphene::chain::pts_address& vo, uint32_t max_depth = 1 );
 }

libraries/chain/include/graphene/chain/tournament_object.hpp

@@ -12,8 +12,8 @@ namespace graphene { namespace chain {
 } }
 
 namespace fc { 
-   void to_variant(const graphene::chain::tournament_object& tournament_obj, fc::variant& v);
+   void to_variant(const graphene::chain::tournament_object& tournament_obj, fc::variant& v, uint32_t max_depth = 1);
-   void from_variant(const fc::variant& v, graphene::chain::tournament_object& tournament_obj);
+   void from_variant(const fc::variant& v, graphene::chain::tournament_object& tournament_obj, uint32_t max_depth = 1);
 } //end namespace fc
 
 namespace graphene { namespace chain {
@@ -108,8 +108,8 @@ namespace graphene { namespace chain {
       template<typename Stream>
       friend Stream& operator>>( Stream& s, tournament_object& tournament_obj );
 
-      friend void ::fc::to_variant(const graphene::chain::tournament_object& tournament_obj, fc::variant& v);
+      friend void ::fc::to_variant(const graphene::chain::tournament_object& tournament_obj, fc::variant& v, uint32_t max_depth);
-      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::tournament_object& tournament_obj);
+      friend void ::fc::from_variant(const fc::variant& v, graphene::chain::tournament_object& tournament_obj, uint32_t max_depth);
 
       void pack_impl(std::ostream& stream) const;
       void unpack_impl(std::istream& stream);

libraries/chain/include/graphene/chain/vesting_balance_object.hpp

@@ -24,8 +24,6 @@
 #pragma once
 
 #include <graphene/chain/protocol/asset.hpp>
-#include <graphene/chain/protocol/vesting.hpp>
-
 #include <graphene/db/object.hpp>
 #include <graphene/db/generic_index.hpp>
 
@@ -35,9 +33,6 @@
 #include <algorithm>
 #include <boost/multi_index/composite_key.hpp>
 
-#define offset_d(i,f)    (long(&(i)->f) - long(i))
-#define offset_s(t,f)    offset_d((t*)1000, f)
-
 namespace graphene { namespace chain {
    using namespace graphene::db;
 
@@ -145,11 +140,12 @@ namespace graphene { namespace chain {
          /// The vesting policy stores details on when funds vest, and controls when they may be withdrawn
          vesting_policy policy;
 
-         /// We can have 2 types of vesting, gpos and all the rest
-         vesting_balance_type balance_type = vesting_balance_type::unspecified;
-
          vesting_balance_object() {}
          
+         asset_id_type get_asset_id() const { return balance.asset_id; }
+
+         share_type get_asset_amount() const { return balance.amount; }
+
          ///@brief Deposit amount into vesting balance, requiring it to vest before withdrawal
          void deposit(const fc::time_point_sec& now, const asset& amount);
          bool is_deposit_allowed(const fc::time_point_sec& now, const asset& amount)const;
@@ -190,14 +186,12 @@ namespace graphene { namespace chain {
            composite_key<
               vesting_balance_object,
               member_offset<vesting_balance_object, asset_id_type, (size_t) (offsetof(vesting_balance_object,balance) + offsetof(asset,asset_id))>,
-              member<vesting_balance_object, vesting_balance_type, &vesting_balance_object::balance_type>,
               member_offset<vesting_balance_object, share_type, (size_t) (offsetof(vesting_balance_object,balance) + offsetof(asset,amount))>
               //member<vesting_balance_object, account_id_type, &vesting_balance_object::owner>
-              //member_offset<vesting_balance_object, account_id_type, (size_t) (offset_s(vesting_balance_object,owner))>
+              //member_offset<vesting_balance_object, account_id_type, (size_t) (offsetof(vesting_balance_object,owner))>
            >,
            composite_key_compare<
               std::less< asset_id_type >,
-              std::less< vesting_balance_type >,
               std::greater< share_type >
               //std::less< account_id_type >
            >
@@ -231,5 +225,4 @@ FC_REFLECT_DERIVED(graphene::chain::vesting_balance_object, (graphene::db::objec
                    (owner)
                    (balance)
                    (policy)
-                   (balance_type)
                   )

libraries/chain/lottery_evaluator.cpp

@@ -0,0 +1,133 @@
+/*
+ * Copyright (c) 2017 Peerplays, Inc., and contributors.
+ *
+ * The MIT License
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#include <graphene/chain/lottery_evaluator.hpp>
+#include <graphene/chain/asset_object.hpp>
+#include <graphene/chain/account_object.hpp>
+#include <graphene/chain/market_object.hpp>
+#include <graphene/chain/database.hpp>
+#include <graphene/chain/exceptions.hpp>
+#include <graphene/chain/hardfork.hpp>
+#include <graphene/chain/is_authorized_asset.hpp>
+
+#include <functional> 
+
+#include <boost/algorithm/string/case_conv.hpp>
+
+namespace graphene { namespace chain {
+
+void_result ticket_purchase_evaluator::do_evaluate( const ticket_purchase_operation& op )
+{ try {
+   lottery = &op.lottery(db());
+   FC_ASSERT( lottery->is_lottery() );
+
+   asset_dynamic_data = &lottery->dynamic_asset_data_id(db());
+   FC_ASSERT( asset_dynamic_data->current_supply < lottery->options.max_supply );
+   FC_ASSERT( (asset_dynamic_data->current_supply.value + op.tickets_to_buy) <= lottery->options.max_supply );
+   
+   auto lottery_options = *lottery->lottery_options;
+   FC_ASSERT( lottery_options.is_active );
+   FC_ASSERT( lottery_options.ticket_price.asset_id == op.amount.asset_id );
+   FC_ASSERT( (double)op.amount.amount.value / lottery_options.ticket_price.amount.value == (double)op.tickets_to_buy );
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+void_result ticket_purchase_evaluator::do_apply( const ticket_purchase_operation& op )
+{ try {
+   db().adjust_balance( op.buyer, -op.amount );
+   db().adjust_balance( op.lottery, op.amount );
+   db().adjust_balance( op.buyer, asset( op.tickets_to_buy, lottery->id ) );
+   db().modify( *asset_dynamic_data, [&]( asset_dynamic_data_object& data ){
+      data.current_supply += op.tickets_to_buy;
+   });
+   db().check_lottery_end_by_participants( op.lottery );
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+void_result lottery_reward_evaluator::do_evaluate( const lottery_reward_operation& op )
+{ try {
+   lottery = &op.lottery(db());
+   FC_ASSERT( lottery->is_lottery() );
+
+   auto lottery_options = *lottery->lottery_options;
+   FC_ASSERT( lottery_options.is_active );
+   FC_ASSERT( db().get_balance(op.lottery).amount > 0 );
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+void_result lottery_reward_evaluator::do_apply( const lottery_reward_operation& op )
+{ try {
+   db().adjust_balance( op.lottery, -op.amount);
+   db().adjust_balance( op.winner, op.amount );
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+
+void_result lottery_end_evaluator::do_evaluate( const lottery_end_operation& op )
+{ try {
+   lottery = &op.lottery(db());
+   FC_ASSERT( lottery->is_lottery() );
+    
+   asset_dynamic_data = &lottery->dynamic_asset_data_id(db());
+   
+   auto lottery_options = *lottery->lottery_options;
+   FC_ASSERT( lottery_options.is_active );
+   FC_ASSERT( db().get_balance(lottery->get_id()).amount == 0 );
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+void_result lottery_end_evaluator::do_apply( const lottery_end_operation& op )
+{ try {
+   db().modify( *asset_dynamic_data, [&]( asset_dynamic_data_object& data ) {
+      data.sweeps_tickets_sold = data.current_supply;
+      data.current_supply = 0;
+   });
+   for( auto account_info : op.participants )
+   {
+      db().adjust_balance( account_info.first, -db().get_balance( account_info.first, op.lottery ) );
+   }
+   db().modify( *lottery, [](asset_object& ao) {
+      ao.lottery_options->is_active = false;
+   });
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+void_result sweeps_vesting_claim_evaluator::do_evaluate( const sweeps_vesting_claim_operation& op )
+{ try {
+   const auto& sweeps_vesting_index = db().get_index_type<sweeps_vesting_balance_index>().indices().get<by_owner>();
+   auto vesting = sweeps_vesting_index.find(op.account);
+   FC_ASSERT( vesting != sweeps_vesting_index.end() );
+   FC_ASSERT( op.amount_to_claim <= vesting->available_for_claim() );
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+void_result sweeps_vesting_claim_evaluator::do_apply( const sweeps_vesting_claim_operation& op )
+{ try {
+   db().adjust_sweeps_vesting_balance( op.account, -op.amount_to_claim.amount.value * SWEEPS_VESTING_BALANCE_MULTIPLIER );
+   db().adjust_balance( op.account, op.amount_to_claim );
+   return void_result();
+} FC_CAPTURE_AND_RETHROW( (op) ) }
+
+
+
+} } // graphene::chain

libraries/chain/match_object.cpp

@@ -364,41 +364,41 @@ namespace graphene { namespace chain {
 
 namespace fc { 
    // Manually reflect match_object to variant to properly reflect "state"
-   void to_variant(const graphene::chain::match_object& match_obj, fc::variant& v)
+   void to_variant(const graphene::chain::match_object& match_obj, fc::variant& v, uint32_t max_depth)
    { try {
       fc_elog(fc::logger::get("tournament"), "In match_obj to_variant");
       elog("In match_obj to_variant");
       fc::mutable_variant_object o;
-      o("id", match_obj.id)
+      o("id", fc::variant(match_obj.id, max_depth))
-       ("tournament_id", match_obj.tournament_id)
+       ("tournament_id", fc::variant(match_obj.tournament_id, max_depth))
-       ("players", match_obj.players)
+       ("players", fc::variant(match_obj.players, max_depth))
-       ("games", match_obj.games)
+       ("games", fc::variant(match_obj.games, max_depth))
-       ("game_winners", match_obj.game_winners)
+       ("game_winners", fc::variant(match_obj.game_winners, max_depth))
-       ("number_of_wins", match_obj.number_of_wins)
+       ("number_of_wins", fc::variant(match_obj.number_of_wins, max_depth))
-       ("number_of_ties", match_obj.number_of_ties)
+       ("number_of_ties", fc::variant(match_obj.number_of_ties, max_depth))
-       ("match_winners", match_obj.match_winners)
+       ("match_winners", fc::variant(match_obj.match_winners, max_depth))
-       ("start_time", match_obj.start_time)
+       ("start_time", fc::variant(match_obj.start_time, max_depth))
-       ("end_time", match_obj.end_time)
+       ("end_time", fc::variant(match_obj.end_time, max_depth))
-       ("state", match_obj.get_state());
+       ("state", fc::variant(match_obj.get_state(), max_depth));
 
       v = o;
    } FC_RETHROW_EXCEPTIONS(warn, "") }
 
    // Manually reflect match_object to variant to properly reflect "state"
-   void from_variant(const fc::variant& v, graphene::chain::match_object& match_obj)
+   void from_variant(const fc::variant& v, graphene::chain::match_object& match_obj, uint32_t max_depth)
    { try {
       fc_elog(fc::logger::get("tournament"), "In match_obj from_variant");
-      match_obj.id = v["id"].as<graphene::chain::match_id_type>();
+      match_obj.id = v["id"].as<graphene::chain::match_id_type>( max_depth );
-      match_obj.tournament_id = v["tournament_id"].as<graphene::chain::tournament_id_type>();
+      match_obj.tournament_id = v["tournament_id"].as<graphene::chain::tournament_id_type>( max_depth );
-      match_obj.players = v["players"].as<std::vector<graphene::chain::account_id_type> >();
+      match_obj.players = v["players"].as<std::vector<graphene::chain::account_id_type> >( max_depth );
-      match_obj.games = v["games"].as<std::vector<graphene::chain::game_id_type> >();
+      match_obj.games = v["games"].as<std::vector<graphene::chain::game_id_type> >( max_depth );
-      match_obj.game_winners = v["game_winners"].as<std::vector<flat_set<graphene::chain::account_id_type> > >();
+      match_obj.game_winners = v["game_winners"].as<std::vector<flat_set<graphene::chain::account_id_type> > >( max_depth );
-      match_obj.number_of_wins = v["number_of_wins"].as<std::vector<uint32_t> >();
+      match_obj.number_of_wins = v["number_of_wins"].as<std::vector<uint32_t> >( max_depth );
-      match_obj.number_of_ties = v["number_of_ties"].as<uint32_t>();
+      match_obj.number_of_ties = v["number_of_ties"].as<uint32_t>( max_depth );
-      match_obj.match_winners = v["match_winners"].as<flat_set<graphene::chain::account_id_type> >();
+      match_obj.match_winners = v["match_winners"].as<flat_set<graphene::chain::account_id_type> >( max_depth );
-      match_obj.start_time = v["start_time"].as<time_point_sec>();
+      match_obj.start_time = v["start_time"].as<time_point_sec>( max_depth );
-      match_obj.end_time = v["end_time"].as<optional<time_point_sec> >();
+      match_obj.end_time = v["end_time"].as<optional<time_point_sec> >( max_depth );
-      graphene::chain::match_state state = v["state"].as<graphene::chain::match_state>();
+      graphene::chain::match_state state = v["state"].as<graphene::chain::match_state>( max_depth );
       const_cast<int*>(match_obj.my->state_machine.current_state())[0] = (int)state;
    } FC_RETHROW_EXCEPTIONS(warn, "") }
 } //end namespace fc

libraries/chain/proposal_evaluator.cpp

@@ -135,11 +135,6 @@ struct proposal_operation_hardfork_visitor
        FC_ASSERT( block_time >= HARDFORK_1000_TIME, "event_update_status_operation not allowed yet!" );
    }
 
-   void operator()(const vesting_balance_create_operation &vbco) const {
-      if(block_time < HARDFORK_GPOS_TIME)
-      FC_ASSERT( vbco.balance_type == vesting_balance_type::unspecified, "balance_type in vesting create not allowed yet!" );
-   }
-
    // loop and self visit in proposals
    void operator()(const proposal_create_operation &v) const {
       for (const op_wrapper &op : v.proposed_ops)
@@ -249,20 +244,6 @@ void_result proposal_update_evaluator::do_evaluate(const proposal_update_operati
                  "", ("id", id)("available", _proposal->available_owner_approvals) );
    }
 
-   /*  All authority checks happen outside of evaluators
-   if( (d.get_node_properties().skip_flags & database::skip_authority_check) == 0 )
-   {
-      for( const auto& id : o.key_approvals_to_add )
-      {
-         FC_ASSERT( trx_state->signed_by(id) );
-      }
-      for( const auto& id : o.key_approvals_to_remove )
-      {
-         FC_ASSERT( trx_state->signed_by(id) );
-      }
-   }
-   */
-
    return void_result();
 } FC_CAPTURE_AND_RETHROW( (o) ) }
 
@@ -298,6 +279,9 @@ void_result proposal_update_evaluator::do_apply(const proposal_update_operation&
       try {
          _processed_transaction = d.push_proposal(*_proposal);
       } catch(fc::exception& e) {
+         d.modify(*_proposal, [&e](proposal_object& p) {
+            p.fail_reason = e.to_string(fc::log_level(fc::log_level::all));
+         });
          wlog("Proposed transaction ${id} failed to apply once approved with exception:\n----\n${reason}\n----\nWill try again when it expires.",
               ("id", o.proposal)("reason", e.to_detail_string()));
          _proposal_failed = true;

libraries/chain/proposal_object.cpp

@@ -43,14 +43,11 @@ bool proposal_object::is_authorized_to_execute(database& db) const
    } 
    catch ( const fc::exception& e )
    {
-      //idump((available_active_approvals));
-      //wlog((e.to_detail_string()));
       return false;
    }
    return true;
 }
 
-
 void required_approval_index::object_inserted( const object& obj )
 {
     assert( dynamic_cast<const proposal_object*>(&obj) );

libraries/chain/protocol/address.cpp

@@ -101,11 +101,11 @@ namespace graphene {
 
 namespace fc
 {
-    void to_variant( const graphene::chain::address& var,  variant& vo )
+    void to_variant( const graphene::chain::address& var,  variant& vo, uint32_t max_depth )
     {
         vo = std::string(var);
     }
-    void from_variant( const variant& var,  graphene::chain::address& vo )
+    void from_variant( const variant& var,  graphene::chain::address& vo, uint32_t max_depth )
     {
         vo = graphene::chain::address( var.as_string() );
     }

libraries/chain/protocol/asset_ops.cpp

@@ -22,6 +22,7 @@
  * THE SOFTWARE.
  */
 #include <graphene/chain/protocol/asset_ops.hpp>
+#include <graphene/chain/database.hpp>
 
 namespace graphene { namespace chain {
 
@@ -77,7 +78,6 @@ share_type asset_issue_operation::calculate_fee(const fee_parameters_type& k)con
 share_type asset_create_operation::calculate_fee(const asset_create_operation::fee_parameters_type& param)const
 {
    auto core_fee_required = param.long_symbol; 
-
    switch(symbol.size()) {
       case 3: core_fee_required = param.symbol3;
          break;
@@ -86,7 +86,6 @@ share_type asset_create_operation::calculate_fee(const asset_create_operation::f
       default:
          break;
    }
-
    // common_options contains several lists and a string. Charge fees for its size
    core_fee_required += calculate_data_fee( fc::raw::pack_size(*this), param.price_per_kbyte );
 
@@ -112,6 +111,35 @@ void  asset_create_operation::validate()const
    FC_ASSERT(precision <= 12);
 }
 
+share_type lottery_asset_create_operation::calculate_fee(const lottery_asset_create_operation::fee_parameters_type& param)const
+{
+   auto core_fee_required = param.lottery_asset; 
+   
+   // common_options contains several lists and a string. Charge fees for its size
+   core_fee_required += calculate_data_fee( fc::raw::pack_size(*this), param.price_per_kbyte );
+
+   return core_fee_required;
+}
+
+void  lottery_asset_create_operation::validate()const
+{
+   FC_ASSERT( fee.amount >= 0 );
+   FC_ASSERT( is_valid_symbol(symbol) );
+   common_options.validate();
+   if( common_options.issuer_permissions & (disable_force_settle|global_settle) )
+      FC_ASSERT( bitasset_opts.valid() );
+   if( is_prediction_market )
+   {
+      FC_ASSERT( bitasset_opts.valid(), "Cannot have a User-Issued Asset implement a prediction market." );
+      FC_ASSERT( common_options.issuer_permissions & global_settle );
+   }
+   if( bitasset_opts ) bitasset_opts->validate();
+
+   asset dummy = asset(1) * common_options.core_exchange_rate;
+   FC_ASSERT(dummy.asset_id == asset_id_type(1));
+   FC_ASSERT(precision <= 12);
+}
+
 void asset_update_operation::validate()const
 {
    FC_ASSERT( fee.amount >= 0 );
@@ -244,4 +272,19 @@ void asset_claim_fees_operation::validate()const {
    FC_ASSERT( amount_to_claim.amount > 0 );
 }
 
+
+void lottery_asset_options::validate() const
+{
+   FC_ASSERT( winning_tickets.size() <= 64 );
+   FC_ASSERT( ticket_price.amount >= 1 );
+   uint16_t total = 0;
+   for( auto benefactor : benefactors ) {
+      total += benefactor.share;
+   }
+   for( auto share : winning_tickets ) {
+      total += share;
+   }
+   FC_ASSERT( total == GRAPHENE_100_PERCENT, "distribution amount not equals GRAPHENE_100_PERCENT" );
+}
+
 } } // namespace graphene::chain

libraries/chain/protocol/fee_schedule.cpp

@@ -124,7 +124,7 @@ namespace graphene { namespace chain {
 
    asset fee_schedule::calculate_fee( const operation& op, const price& core_exchange_rate )const
    {
-      //idump( (op)(core_exchange_rate) );
+      //+( (op)(core_exchange_rate) );
       fee_parameters params; params.set_which(op.which());
       auto itr = parameters.find(params);
       if( itr != parameters.end() ) params = *itr;

libraries/chain/protocol/lottery_ops.cpp

@@ -0,0 +1,39 @@
+/*
+ * Copyright (c) 2017 Peerplays, Inc., and contributors.
+ *
+ * The MIT License
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#include <graphene/chain/protocol/lottery_ops.hpp>
+
+namespace graphene { namespace chain {
+
+void ticket_purchase_operation::validate() const 
+{
+   FC_ASSERT( fee.amount >= 0 );
+   FC_ASSERT( tickets_to_buy > 0 );
+}
+
+share_type ticket_purchase_operation::calculate_fee( const fee_parameters_type& k )const
+{
+   return k.fee;
+}
+
+} } // namespace graphene::chain

libraries/chain/protocol/transaction.cpp

@@ -71,6 +71,7 @@ const signature_type& graphene::chain::signed_transaction::sign(const private_ke
 {
    digest_type h = sig_digest( chain_id );
    signatures.push_back(key.sign_compact(h));
+   signees.clear(); // Clear signees since it may be inconsistent after added a new signature
    return signatures.back();
 }
 
@@ -297,8 +298,12 @@ void verify_authority( const vector<operation>& ops, const flat_set<public_key_t
 } FC_CAPTURE_AND_RETHROW( (ops)(sigs) ) }
 
 
-flat_set<public_key_type> signed_transaction::get_signature_keys( const chain_id_type& chain_id )const
+const flat_set<public_key_type>& signed_transaction::get_signature_keys( const chain_id_type& chain_id )const
 { try {
+   // Strictly we should check whether the given chain ID is same as the one used to initialize the `signees` field.
+   // However, we don't pass in another chain ID so far, for better performance, we skip the check.
+   if( signees.empty() && !signatures.empty() )
+   {
       auto d = sig_digest( chain_id );
       flat_set<public_key_type> result;
       for( const auto&  sig : signatures )
@@ -308,11 +313,12 @@ flat_set<public_key_type> signed_transaction::get_signature_keys( const chain_id
             tx_duplicate_sig,
             "Duplicate Signature detected" );
       }
-   return result;
+      signees = std::move( result );
+   }
+   return signees;
 } FC_CAPTURE_AND_RETHROW() }
 
 
-
 set<public_key_type> signed_transaction::get_required_signatures(
    const chain_id_type& chain_id,
    const flat_set<public_key_type>& available_keys,
@@ -325,8 +331,8 @@ set<public_key_type> signed_transaction::get_required_signatures(
    vector<authority> other;
    get_required_authorities( required_active, required_owner, other );
 
-
+   const flat_set<public_key_type>& signature_keys = get_signature_keys( chain_id );
-   sign_state s(get_signature_keys( chain_id ),get_active,available_keys);
+   sign_state s( signature_keys, get_active, available_keys );
    s.max_recursion = max_recursion_depth;
 
    for( const auto& auth : other )

libraries/chain/protocol/types.cpp

@@ -248,32 +248,32 @@ namespace graphene { namespace chain {
 namespace fc
 {
     using namespace std;
-    void to_variant( const graphene::chain::public_key_type& var,  fc::variant& vo )
+    void to_variant( const graphene::chain::public_key_type& var,  fc::variant& vo, uint32_t max_depth )
     {
         vo = std::string( var );
     }
 
-    void from_variant( const fc::variant& var,  graphene::chain::public_key_type& vo )
+    void from_variant( const fc::variant& var,  graphene::chain::public_key_type& vo, uint32_t max_depth )
     {
         vo = graphene::chain::public_key_type( var.as_string() );
     }
     
-    void to_variant( const graphene::chain::extended_public_key_type& var, fc::variant& vo )
+    void to_variant( const graphene::chain::extended_public_key_type& var, fc::variant& vo, uint32_t max_depth )
     {
        vo = std::string( var );
     }
     
-    void from_variant( const fc::variant& var, graphene::chain::extended_public_key_type& vo )
+    void from_variant( const fc::variant& var, graphene::chain::extended_public_key_type& vo, uint32_t max_depth )
     {
        vo = graphene::chain::extended_public_key_type( var.as_string() );
     }
 
-    void to_variant( const graphene::chain::extended_private_key_type& var, fc::variant& vo )
+    void to_variant( const graphene::chain::extended_private_key_type& var, fc::variant& vo, uint32_t max_depth )
     {
        vo = std::string( var );
     }
     
-    void from_variant( const fc::variant& var, graphene::chain::extended_private_key_type& vo )
+    void from_variant( const fc::variant& var, graphene::chain::extended_private_key_type& vo, uint32_t max_depth )
     {
        vo = graphene::chain::extended_private_key_type( var.as_string() );
     }

libraries/chain/protocol/vote.cpp

@@ -38,12 +38,12 @@ vote_id_type get_next_vote_id( global_property_object& gpo, vote_id_type::vote_t
 namespace fc
 {
 
-void to_variant(const graphene::chain::vote_id_type& var, variant& vo)
+void to_variant( const graphene::chain::vote_id_type& var, variant& vo, uint32_t max_depth )
 {
    vo = string(var);
 }
 
-void from_variant(const variant& var, graphene::chain::vote_id_type& vo)
+void from_variant( const variant& var, graphene::chain::vote_id_type& vo, uint32_t max_depth )
 {
    vo = graphene::chain::vote_id_type(var.as_string());
 }

libraries/chain/pts_address.cpp

@@ -89,11 +89,11 @@ namespace graphene { namespace chain {
 
 namespace fc
 {
-   void to_variant( const graphene::chain::pts_address& var,  variant& vo )
+   void to_variant( const graphene::chain::pts_address& var,  variant& vo, uint32_t max_depth )
    {
         vo = std::string(var);
    }
-   void from_variant( const variant& var,  graphene::chain::pts_address& vo )
+   void from_variant( const variant& var,  graphene::chain::pts_address& vo, uint32_t max_depth )
    {
         vo = graphene::chain::pts_address( var.as_string() );
    }

libraries/chain/tournament_object.cpp

@@ -722,37 +722,37 @@ namespace graphene { namespace chain {
 
 namespace fc { 
    // Manually reflect tournament_object to variant to properly reflect "state"
-   void to_variant(const graphene::chain::tournament_object& tournament_obj, fc::variant& v)
+   void to_variant(const graphene::chain::tournament_object& tournament_obj, fc::variant& v, uint32_t max_depth)
    {
       fc_elog(fc::logger::get("tournament"), "In tournament_obj to_variant");
       elog("In tournament_obj to_variant");
       fc::mutable_variant_object o;
-      o("id", tournament_obj.id)
+      o("id", fc::variant(tournament_obj.id, max_depth))
-       ("creator", tournament_obj.creator)
+       ("creator", fc::variant(tournament_obj.creator, max_depth))
-       ("options", tournament_obj.options)
+       ("options", fc::variant(tournament_obj.options, max_depth))
-       ("start_time", tournament_obj.start_time)
+       ("start_time", fc::variant(tournament_obj.start_time, max_depth))
-       ("end_time", tournament_obj.end_time)
+       ("end_time", fc::variant(tournament_obj.end_time, max_depth))
-       ("prize_pool", tournament_obj.prize_pool)
+       ("prize_pool", fc::variant(tournament_obj.prize_pool, max_depth))
-       ("registered_players", tournament_obj.registered_players)
+       ("registered_players", fc::variant(tournament_obj.registered_players, max_depth))
-       ("tournament_details_id", tournament_obj.tournament_details_id)
+       ("tournament_details_id", fc::variant(tournament_obj.tournament_details_id, max_depth))
-       ("state", tournament_obj.get_state());
+       ("state", fc::variant(tournament_obj.get_state(), max_depth));
 
       v = o;
    }
 
    // Manually reflect tournament_object to variant to properly reflect "state"
-   void from_variant(const fc::variant& v, graphene::chain::tournament_object& tournament_obj)
+   void from_variant(const fc::variant& v, graphene::chain::tournament_object& tournament_obj, uint32_t max_depth)
    {
       fc_elog(fc::logger::get("tournament"), "In tournament_obj from_variant");
-      tournament_obj.id = v["id"].as<graphene::chain::tournament_id_type>();
+      tournament_obj.id = v["id"].as<graphene::chain::tournament_id_type>( max_depth );
-      tournament_obj.creator = v["creator"].as<graphene::chain::account_id_type>();
+      tournament_obj.creator = v["creator"].as<graphene::chain::account_id_type>( max_depth );
-      tournament_obj.options = v["options"].as<graphene::chain::tournament_options>();
+      tournament_obj.options = v["options"].as<graphene::chain::tournament_options>( max_depth );
-      tournament_obj.start_time = v["start_time"].as<optional<time_point_sec> >();
+      tournament_obj.start_time = v["start_time"].as<optional<time_point_sec> >( max_depth );
-      tournament_obj.end_time = v["end_time"].as<optional<time_point_sec> >();
+      tournament_obj.end_time = v["end_time"].as<optional<time_point_sec> >( max_depth );
-      tournament_obj.prize_pool = v["prize_pool"].as<graphene::chain::share_type>();
+      tournament_obj.prize_pool = v["prize_pool"].as<graphene::chain::share_type>( max_depth );
-      tournament_obj.registered_players = v["registered_players"].as<uint32_t>();
+      tournament_obj.registered_players = v["registered_players"].as<uint32_t>( max_depth );
-      tournament_obj.tournament_details_id = v["tournament_details_id"].as<graphene::chain::tournament_details_id_type>();
+      tournament_obj.tournament_details_id = v["tournament_details_id"].as<graphene::chain::tournament_details_id_type>( max_depth );
-      graphene::chain::tournament_state state = v["state"].as<graphene::chain::tournament_state>();
+      graphene::chain::tournament_state state = v["state"].as<graphene::chain::tournament_state>( max_depth );
       const_cast<int*>(tournament_obj.my->state_machine.current_state())[0] = (int)state;
    }
 } //end namespace fc

libraries/chain/vesting_balance_evaluator.cpp

@@ -42,9 +42,6 @@ void_result vesting_balance_create_evaluator::do_evaluate( const vesting_balance
    FC_ASSERT( d.get_balance( creator_account.id, op.amount.asset_id ) >= op.amount );
    FC_ASSERT( !op.amount.asset_id(d).is_transfer_restricted() );
 
-   if(d.head_block_time() < HARDFORK_GPOS_TIME) // Todo: can be removed after gpos hf time pass
-      FC_ASSERT( op.balance_type == vesting_balance_type::unspecified);
-
    return void_result();
 } FC_CAPTURE_AND_RETHROW( (op) ) }
 
@@ -95,20 +92,7 @@ object_id_type vesting_balance_create_evaluator::do_apply( const vesting_balance
       // If making changes to this logic, check if those changes should also be made there as well.
       obj.owner = op.owner;
       obj.balance = op.amount;
-      if(op.balance_type == vesting_balance_type::gpos)
+      op.policy.visit( init_policy_visitor( obj.policy, op.amount.amount, now ) );
-      {
-         const auto &gpo = d.get_global_properties();
-         // forcing gpos policy
-         linear_vesting_policy p;
-         p.begin_timestamp = now;
-         p.vesting_cliff_seconds = gpo.parameters.gpos_subperiod();
-         p.vesting_duration_seconds = gpo.parameters.gpos_subperiod();
-         obj.policy = p;
-      }
-      else {
-         op.policy.visit(init_policy_visitor(obj.policy, op.amount.amount, now));
-      }
-      obj.balance_type = op.balance_type;
    } );
 
 

libraries/chain/witness_evaluator.cpp

@@ -43,10 +43,11 @@ object_id_type witness_create_evaluator::do_apply( const witness_create_operatio
       vote_id = get_next_vote_id(p, vote_id_type::witness);
    });
 
-   const auto& new_witness_object = db().create<witness_object>( [&]( witness_object& obj ){
+   const auto& new_witness_object = db().create<witness_object>( [&]( witness_object& obj ) {
          obj.witness_account  = op.witness_account;
          obj.signing_key      = op.block_signing_key;
-         obj.next_secret_hash = op.initial_secret;
+         obj.previous_secret  = secret_hash_type();
+         obj.next_secret_hash = secret_hash_type::hash( op.initial_secret );
          obj.vote_id          = vote_id;
          obj.url              = op.url;
    });

libraries/db/include/graphene/db/index.hpp

@@ -23,11 +23,13 @@
  */
 #pragma once
 #include <graphene/db/object.hpp>
+
 #include <fc/interprocess/file_mapping.hpp>
 #include <fc/io/raw.hpp>
 #include <fc/io/json.hpp>
 #include <fc/crypto/sha256.hpp>
 #include <fstream>
+#include <stack>
 
 namespace graphene { namespace db {
    class object_database;
@@ -130,7 +132,7 @@ namespace graphene { namespace db {
          virtual fc::uint128        hash()const = 0;
          virtual void               add_observer( const shared_ptr<index_observer>& ) = 0;
 
-         virtual void               object_from_variant( const fc::variant& var, object& obj )const = 0;
+         virtual void               object_from_variant( const fc::variant& var, object& obj, uint32_t max_depth )const = 0;
          virtual void               object_default( object& obj )const = 0;
    };
 
@@ -190,6 +192,111 @@ namespace graphene { namespace db {
          object_database& _db;
    };
 
+   /** @class direct_index
+    *  @brief A secondary index that tracks objects in vectors indexed by object
+    *  id. It is meant for fully (or almost fully) populated indexes only (will
+    *  fail when loading an object_database with large gaps).
+    *
+    *  WARNING! If any of the methods called on insertion, removal or
+    *  modification throws, subsequent behaviour is undefined! Such exceptions
+    *  indicate that this index type is not appropriate for the use-case.
+    */
+   template<typename Object, uint8_t chunkbits>
+   class direct_index : public secondary_index
+   {
+      static_assert( chunkbits < 64, "Do you really want arrays with more than 2^63 elements???" );
+
+      // private
+         static const size_t MAX_HOLE = 100;
+         static const size_t _mask = ((1 << chunkbits) - 1);
+         size_t next = 0;
+         vector< vector< const Object* > > content;
+         std::stack< object_id_type > ids_being_modified;
+
+      public:
+         direct_index() {
+            FC_ASSERT( (1ULL << chunkbits) > MAX_HOLE, "Small chunkbits is inefficient." );
+         }
+
+         virtual ~direct_index(){}
+
+         virtual void object_inserted( const object& obj )
+         {
+            uint64_t instance = obj.id.instance();
+            if( instance == next )
+            {
+               if( !(next & _mask) )
+               {
+                  content.resize((next >> chunkbits) + 1);
+                  content[next >> chunkbits].resize( 1 << chunkbits, nullptr );
+               }
+               next++;
+            }
+            else if( instance < next )
+               FC_ASSERT( !content[instance >> chunkbits][instance & _mask], "Overwriting insert at {id}!", ("id",obj.id) );
+            else // instance > next, allow small "holes"
+            {
+               FC_ASSERT( instance <= next + MAX_HOLE, "Out-of-order insert: {id} > {next}!", ("id",obj.id)("next",next) );
+               if( !(next & _mask) || (next & (~_mask)) != (instance & (~_mask)) )
+               {
+                  content.resize((instance >> chunkbits) + 1);
+                  content[instance >> chunkbits].resize( 1 << chunkbits, nullptr );
+               }
+               while( next <= instance )
+               {
+                  content[next >> chunkbits][next & _mask] = nullptr;
+                  next++;
+               }
+            }
+            FC_ASSERT( nullptr != dynamic_cast<const Object*>(&obj), "Wrong object type!" );
+            content[instance >> chunkbits][instance & _mask] = static_cast<const Object*>( &obj );
+         }
+
+         virtual void object_removed( const object& obj )
+         {
+            FC_ASSERT( nullptr != dynamic_cast<const Object*>(&obj), "Wrong object type!" );
+            uint64_t instance = obj.id.instance();
+            FC_ASSERT( instance < next, "Removing out-of-range object: {id} > {next}!", ("id",obj.id)("next",next) );
+            FC_ASSERT( content[instance >> chunkbits][instance & _mask], "Removing non-existent object {id}!", ("id",obj.id) );
+            content[instance >> chunkbits][instance & _mask] = nullptr;
+         }
+
+         virtual void about_to_modify( const object& before )
+         {
+            ids_being_modified.emplace( before.id );
+         }
+
+         virtual void object_modified( const object& after  )
+         {
+            FC_ASSERT( ids_being_modified.top() == after.id, "Modification of ID is not supported!");
+            ids_being_modified.pop();
+         }
+
+         template< typename object_id >
+         const Object* find( const object_id& id )const
+         {
+            static_assert( object_id::space_id == Object::space_id, "Space ID mismatch!" );
+            static_assert( object_id::type_id == Object::type_id, "Type_ID mismatch!" );
+            if( id.instance >= next ) return nullptr;
+            return content[id.instance.value >> chunkbits][id.instance.value & _mask];
+         };
+
+         template< typename object_id >
+         const Object& get( const object_id& id )const
+         {
+            const Object* ptr = find( id );
+            FC_ASSERT( ptr != nullptr, "Object not found!" );
+            return *ptr;
+         };
+
+         const Object* find( const object_id_type& id )const
+         {
+            FC_ASSERT( id.space() == Object::space_id, "Space ID mismatch!" );
+            FC_ASSERT( id.type() == Object::type_id, "Type_ID mismatch!" );
+            if( id.instance() >= next ) return nullptr;
+            return content[id.instance() >> chunkbits][id.instance() & ((1 << chunkbits) - 1)];
+         };
+   };
    
    /**
     * @class primary_index
@@ -198,14 +305,18 @@ namespace graphene { namespace db {
     *
     *  @see http://en.wikipedia.org/wiki/Curiously_recurring_template_pattern
     */
-   template<typename DerivedIndex>
+   template<typename DerivedIndex, uint8_t DirectBits = 0>
    class primary_index  : public DerivedIndex, public base_primary_index
    {
       public:
          typedef typename DerivedIndex::object_type object_type;
 
          primary_index( object_database& db )
-         :base_primary_index(db),_next_id(object_type::space_id,object_type::type_id,0) {}
+         :base_primary_index(db),_next_id(object_type::space_id,object_type::type_id,0)
+         {
+            if( DirectBits > 0 )
+               _direct_by_id = add_secondary_index< direct_index< object_type, DirectBits > >();
+         }
 
          virtual uint8_t object_space_id()const override
          { return object_type::space_id; }
@@ -217,6 +328,14 @@ namespace graphene { namespace db {
          virtual void           use_next_id()override                    { ++_next_id.number;  }
          virtual void           set_next_id( object_id_type id )override { _next_id = id;      }
          
+         /** @return the object with id or nullptr if not found */
+         virtual const object*  find( object_id_type id )const override
+         {
+            if( DirectBits > 0 )
+               return _direct_by_id->find( id );
+            return DerivedIndex::find( id );
+         }
+         
          fc::sha256 get_object_version()const
          {
             std::string desc = "1.0";//get_type_description<object_type>();
@@ -234,14 +353,12 @@ namespace graphene { namespace db {
             fc::raw::unpack(ds, _next_id);
             fc::raw::unpack(ds, open_ver);
             FC_ASSERT( open_ver == get_object_version(), "Incompatible Version, the serialization of objects in this index has changed" );
-            try {
             vector<char> tmp;
-               while( true ) 
+            while( ds.remaining() > 0 ) 
             {
                fc::raw::unpack( ds, tmp );
                load( tmp );
             }
-            } catch ( const fc::exception&  ){}
          }
 
          virtual void save( const path& db ) override 
@@ -301,12 +418,12 @@ namespace graphene { namespace db {
             _observers.emplace_back( o );
          }
 
-         virtual void object_from_variant( const fc::variant& var, object& obj )const override
+         virtual void object_from_variant( const fc::variant& var, object& obj, uint32_t max_depth )const override
          {
             object_id_type id = obj.id;
             object_type* result = dynamic_cast<object_type*>( &obj );
             FC_ASSERT( result != nullptr );
-            fc::from_variant( var, *result );
+            fc::from_variant( var, *result, max_depth );
             obj.id = id;
          }
 
@@ -321,6 +438,7 @@ namespace graphene { namespace db {
 
       private:
          object_id_type                                 _next_id;
+         const direct_index< object_type, DirectBits >* _direct_by_id = nullptr;
    };
 
 } } // graphene::db

libraries/db/include/graphene/db/object.hpp

@@ -27,6 +27,8 @@
 #include <fc/crypto/city.hpp>
 #include <fc/uint128.hpp>
 
+#define MAX_NESTING (200)
+
 namespace graphene { namespace db {
 
    /**
@@ -98,7 +100,7 @@ namespace graphene { namespace db {
          {
             static_cast<DerivedClass&>(*this) = std::move( static_cast<DerivedClass&>(obj) );
          }
-         virtual variant to_variant()const { return variant( static_cast<const DerivedClass&>(*this) ); }
+         virtual variant to_variant()const { return variant( static_cast<const DerivedClass&>(*this), MAX_NESTING ); }
          virtual vector<char> pack()const  { return fc::raw::pack( static_cast<const DerivedClass&>(*this) ); }
          virtual fc::uint128  hash()const  {  
              auto tmp = this->pack();

libraries/db/include/graphene/db/object_id.hpp

@@ -169,12 +169,12 @@ struct reflector<graphene::db::object_id<SpaceID,TypeID,T> >
 };
 
 
- inline void to_variant( const graphene::db::object_id_type& var,  fc::variant& vo )
+ inline void to_variant( const graphene::db::object_id_type& var,  fc::variant& vo, uint32_t max_depth = 1 )
  {
     vo = std::string( var );
  }
 
- inline void from_variant( const fc::variant& var,  graphene::db::object_id_type& vo )
+ inline void from_variant( const fc::variant& var,  graphene::db::object_id_type& vo, uint32_t max_depth = 1 )
  { try {
     vo.number = 0;
     const auto& s = var.get_string();
@@ -191,12 +191,12 @@ struct reflector<graphene::db::object_id<SpaceID,TypeID,T> >
     vo.number |= (space_id << 56) | (type_id << 48);
  } FC_CAPTURE_AND_RETHROW( (var) ) }
  template<uint8_t SpaceID, uint8_t TypeID, typename T>
- void to_variant( const graphene::db::object_id<SpaceID,TypeID,T>& var,  fc::variant& vo )
+ void to_variant( const graphene::db::object_id<SpaceID,TypeID,T>& var,  fc::variant& vo, uint32_t max_depth = 1 )
  {
     vo = fc::to_string(SpaceID) + "." + fc::to_string(TypeID) + "." + fc::to_string(var.instance.value);
  }
  template<uint8_t SpaceID, uint8_t TypeID, typename T>
- void from_variant( const fc::variant& var,  graphene::db::object_id<SpaceID,TypeID,T>& vo )
+ void from_variant( const fc::variant& var,  graphene::db::object_id<SpaceID,TypeID,T>& vo, uint32_t max_depth = 1 )
  { try {
     const auto& s = var.get_string();
     auto first_dot = s.find('.');

libraries/db/object_database.cpp

@@ -71,14 +71,20 @@ index& object_database::get_mutable_index(uint8_t space_id, uint8_t type_id)
 void object_database::flush()
 {
 //   ilog("Save object_database in ${d}", ("d", _data_dir));
+   fc::create_directories( _data_dir / "object_database.tmp" / "lock" );
    for( uint32_t space = 0; space < _index.size(); ++space )
    {
-      fc::create_directories( _data_dir / "object_database" / fc::to_string(space) );
+      fc::create_directories( _data_dir / "object_database.tmp" / fc::to_string(space) );
       const auto types = _index[space].size();
       for( uint32_t type = 0; type  <  types; ++type )
          if( _index[space][type] )
-            _index[space][type]->save( _data_dir / "object_database" / fc::to_string(space)/fc::to_string(type) );
+            _index[space][type]->save( _data_dir / "object_database.tmp" / fc::to_string(space)/fc::to_string(type) );
    }
+   fc::remove_all( _data_dir / "object_database.tmp" / "lock" );
+   if( fc::exists( _data_dir / "object_database" ) )
+      fc::rename( _data_dir / "object_database", _data_dir / "object_database.old" );
+   fc::rename( _data_dir / "object_database.tmp", _data_dir / "object_database" );
+   fc::remove_all( _data_dir / "object_database.old" );
 }
 
 void object_database::wipe(const fc::path& data_dir)
@@ -91,8 +97,13 @@ void object_database::wipe(const fc::path& data_dir)
 
 void object_database::open(const fc::path& data_dir)
 { try {
-   ilog("Opening object database from ${d} ...", ("d", data_dir));
    _data_dir = data_dir;
+   if( fc::exists( _data_dir / "object_database" / "lock" ) )
+   {
+       wlog("Ignoring locked object_database");
+       return;
+   }
+   ilog("Opening object database from ${d} ...", ("d", data_dir));
    for( uint32_t space = 0; space < _index.size(); ++space )
       for( uint32_t type = 0; type  < _index[space].size(); ++type )
          if( _index[space][type] )

libraries/db/undo_database.cpp

@@ -118,8 +118,6 @@ void undo_database::undo()
       _db.insert( std::move(*item.second) );
 
    _stack.pop_back();
-   if( _stack.empty() )
-      _stack.emplace_back();
    enable();
    --_active_sessions;
 } FC_CAPTURE_AND_RETHROW() }
@@ -127,6 +125,12 @@ void undo_database::undo()
 void undo_database::merge()
 {
    FC_ASSERT( _active_sessions > 0 );
+   if( _active_sessions == 1 && _stack.size() == 1 )
+   {
+      _stack.pop_back();
+      --_active_sessions;
+      return;
+   }
    FC_ASSERT( _stack.size() >=2 );
    auto& state = _stack.back();
    auto& prev_state = _stack[_stack.size()-2];

libraries/deterministic_openssl_rand/CMakeLists.txt

@@ -0,0 +1,28 @@
+
+file(GLOB headers "include/graphene/utilities/*.hpp")
+
+set(sources deterministic_openssl_rand.cpp
+            ${headers})
+
+add_library( deterministic_openssl_rand 
+             ${sources}
+             ${HEADERS} )
+target_link_libraries( deterministic_openssl_rand fc )
+target_include_directories( deterministic_openssl_rand
+                            PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include"
+                            "${CMAKE_CURRENT_SOURCE_DIR}/../blockchain/include"
+                          )
+
+if (USE_PCH)
+  set_target_properties(deterministic_openssl_rand PROPERTIES COTIRE_ADD_UNITY_BUILD FALSE)
+  cotire(deterministic_openssl_rand)
+endif(USE_PCH)
+
+install( TARGETS
+   deterministic_openssl_rand
+
+   RUNTIME DESTINATION bin
+   LIBRARY DESTINATION lib
+   ARCHIVE DESTINATION lib
+)
+install( FILES ${headers} DESTINATION "include/graphene/deterministic_openssl_rand" )

libraries/egenesis/egenesis_brief.cpp.tmpl

@@ -26,7 +26,7 @@ using namespace graphene::chain;
 
 chain_id_type get_egenesis_chain_id()
 {
-   return chain_id_type( "${chain_id}$" );
+   return chain_id_type( "${chain_id}" );
 }
 
 void compute_egenesis_json( std::string& result )

libraries/egenesis/egenesis_full.cpp.tmpl

@@ -24,26 +24,25 @@ namespace graphene { namespace egenesis {
 
 using namespace graphene::chain;
 
-static const char genesis_json_array[${genesis_json_array_height}$][${genesis_json_array_width}$+1] =
+static const char genesis_json_array[${genesis_json_array_height}][${genesis_json_array_width}+1] =
 {
-${genesis_json_array}$
+${genesis_json_array}
 };
 
 chain_id_type get_egenesis_chain_id()
 {
-   return chain_id_type( "${chain_id}$" );
+   return chain_id_type( "${chain_id}" );
 }
 
 void compute_egenesis_json( std::string& result )
 {
-   result.reserve( ${genesis_json_length}$ );
+   result.reserve( ${genesis_json_length} );
    result.resize(0);
-   for( size_t i=0; i<${genesis_json_array_height}$-1; i++ )
+   for( size_t i=0; i<${genesis_json_array_height}-1; i++ )
    {
-      result.append( genesis_json_array[i], ${genesis_json_array_width}$ );
+      result.append( genesis_json_array[i], ${genesis_json_array_width} );
    }
-   result.append( std::string( genesis_json_array[ ${genesis_json_array_height}$-1 ] ) );
+   result.append( std::string( genesis_json_array[ ${genesis_json_array_height}-1 ] ) );
-   return;
 }
 
 fc::sha256 get_egenesis_json_hash()

libraries/egenesis/embed_genesis.cpp

@@ -168,7 +168,7 @@ struct egenesis_info
          // If genesis not exist, generate from genesis_json
          try
          {
-            genesis = fc::json::from_string( *genesis_json ).as< genesis_state_type >();
+            genesis = fc::json::from_string( *genesis_json ).as< genesis_state_type >( 20 );
          }
          catch (const fc::exception& e)
          {
@@ -223,7 +223,6 @@ void load_genesis(
       std::cerr << "embed_genesis:  Genesis ID from argument is " << chain_id_str << "\n";
       info.chain_id = chain_id_str;
    }
-   return;
 }
 
 int main( int argc, char** argv )

libraries/fc

@@ -1 +1 @@
-Subproject commit 94b046dce6bb86fd22abd1831fc9056103f4aa5d
+Subproject commit f13d0632b08b9983a275304317a033914938e339

libraries/net/include/graphene/net/config.hpp

@@ -106,3 +106,7 @@
 #define GRAPHENE_NET_MIN_BLOCK_IDS_TO_PREFETCH               10000
 
 #define GRAPHENE_NET_MAX_TRX_PER_SECOND                      1000
+
+#define GRAPHENE_NET_MAX_NESTED_OBJECTS                      (250)
+
+#define MAXIMUM_PEERDB_SIZE 1000

libraries/net/node.cpp

@@ -1260,13 +1260,11 @@ namespace graphene { namespace net { namespace detail {
             wdump((inventory_to_advertise));
             for (const item_id& item_to_advertise : inventory_to_advertise)
             {
-              if (peer->inventory_advertised_to_peer.find(item_to_advertise) != peer->inventory_advertised_to_peer.end() )
+               auto adv_to_peer = peer->inventory_advertised_to_peer.find(item_to_advertise);
-                 wdump((*peer->inventory_advertised_to_peer.find(item_to_advertise)));
+               auto adv_to_us   = peer->inventory_peer_advertised_to_us.find(item_to_advertise);
-              if (peer->inventory_peer_advertised_to_us.find(item_to_advertise) != peer->inventory_peer_advertised_to_us.end() )
-                 wdump((*peer->inventory_peer_advertised_to_us.find(item_to_advertise)));
 
-              if (peer->inventory_advertised_to_peer.find(item_to_advertise) == peer->inventory_advertised_to_peer.end() &&
+              if (adv_to_peer == peer->inventory_advertised_to_peer.end() &&
-                  peer->inventory_peer_advertised_to_us.find(item_to_advertise) == peer->inventory_peer_advertised_to_us.end())
+                  adv_to_us == peer->inventory_peer_advertised_to_us.end())
               {
                 items_to_advertise_by_type[item_to_advertise.item_type].push_back(item_to_advertise.item_hash);
                 peer->inventory_advertised_to_peer.insert(peer_connection::timestamped_item_id(item_to_advertise, fc::time_point::now()));
@@ -1275,6 +1273,13 @@ namespace graphene { namespace net { namespace detail {
                   testnetlog("advertising transaction ${id} to peer ${endpoint}", ("id", item_to_advertise.item_hash)("endpoint", peer->get_remote_endpoint()));
                 dlog("advertising item ${id} to peer ${endpoint}", ("id", item_to_advertise.item_hash)("endpoint", peer->get_remote_endpoint()));
               }
+              else
+              {
+                 if (adv_to_peer != peer->inventory_advertised_to_peer.end() )
+                    wdump( (*adv_to_peer) );
+                 if (adv_to_us != peer->inventory_peer_advertised_to_us.end() )
+                    wdump( (*adv_to_us) );
+              }
             }
               dlog("advertising ${count} new item(s) of ${types} type(s) to peer ${endpoint}",
                    ("count", total_items_to_send_to_this_peer)
@@ -1853,10 +1858,10 @@ namespace graphene { namespace net { namespace detail {
 #endif
       user_data["bitness"] = sizeof(void*) * 8;
 
-      user_data["node_id"] = _node_id;
+      user_data["node_id"] = fc::variant( _node_id, 1 );
 
       item_hash_t head_block_id = _delegate->get_head_block_id();
-      user_data["last_known_block_hash"] = head_block_id;
+      user_data["last_known_block_hash"] = fc::variant( head_block_id, 1 );
       user_data["last_known_block_number"] = _delegate->get_block_number(head_block_id);
       user_data["last_known_block_time"] = _delegate->get_block_time(head_block_id);
 
@@ -1872,19 +1877,19 @@ namespace graphene { namespace net { namespace detail {
       if (user_data.contains("graphene_git_revision_sha"))
         originating_peer->graphene_git_revision_sha = user_data["graphene_git_revision_sha"].as_string();
       if (user_data.contains("graphene_git_revision_unix_timestamp"))
-        originating_peer->graphene_git_revision_unix_timestamp = fc::time_point_sec(user_data["graphene_git_revision_unix_timestamp"].as<uint32_t>());
+        originating_peer->graphene_git_revision_unix_timestamp = fc::time_point_sec(user_data["graphene_git_revision_unix_timestamp"].as<uint32_t>(1));
       if (user_data.contains("fc_git_revision_sha"))
         originating_peer->fc_git_revision_sha = user_data["fc_git_revision_sha"].as_string();
       if (user_data.contains("fc_git_revision_unix_timestamp"))
-        originating_peer->fc_git_revision_unix_timestamp = fc::time_point_sec(user_data["fc_git_revision_unix_timestamp"].as<uint32_t>());
+        originating_peer->fc_git_revision_unix_timestamp = fc::time_point_sec(user_data["fc_git_revision_unix_timestamp"].as<uint32_t>(1));
       if (user_data.contains("platform"))
         originating_peer->platform = user_data["platform"].as_string();
       if (user_data.contains("bitness"))
-        originating_peer->bitness = user_data["bitness"].as<uint32_t>();
+        originating_peer->bitness = user_data["bitness"].as<uint32_t>(1);
       if (user_data.contains("node_id"))
-        originating_peer->node_id = user_data["node_id"].as<node_id_t>();
+        originating_peer->node_id = user_data["node_id"].as<node_id_t>(1);
       if (user_data.contains("last_known_fork_block_number"))
-        originating_peer->last_known_fork_block_number = user_data["last_known_fork_block_number"].as<uint32_t>();
+        originating_peer->last_known_fork_block_number = user_data["last_known_fork_block_number"].as<uint32_t>(1);
     }
 
     void node_impl::on_hello_message( peer_connection* originating_peer, const hello_message& hello_message_received )
@@ -1894,7 +1899,7 @@ namespace graphene { namespace net { namespace detail {
       node_id_t peer_node_id = hello_message_received.node_public_key;
       try
       {
-        peer_node_id = hello_message_received.user_data["node_id"].as<node_id_t>();
+        peer_node_id = hello_message_received.user_data["node_id"].as<node_id_t>(1);
       }
       catch (const fc::exception&)
       {
@@ -2935,7 +2940,7 @@ namespace graphene { namespace net { namespace detail {
              ( "msg", closing_connection_message_received.reason_for_closing )
              ( "error", closing_connection_message_received.error ) );
         std::ostringstream message;
-        message << "Peer " << fc::variant( originating_peer->get_remote_endpoint() ).as_string() <<
+        message << "Peer " << fc::variant( originating_peer->get_remote_endpoint(), GRAPHENE_NET_MAX_NESTED_OBJECTS ).as_string() <<
                   " disconnected us: " << closing_connection_message_received.reason_for_closing;
         fc::exception detailed_error(FC_LOG_MESSAGE(warn, "Peer ${peer} is disconnecting us because of an error: ${msg}, exception: ${error}",
                                                     ( "peer", originating_peer->get_remote_endpoint() )
@@ -3841,7 +3846,7 @@ namespace graphene { namespace net { namespace detail {
           user_data["bitness"] = *peer->bitness;
         user_data["user_agent"] = peer->user_agent;
 
-        user_data["last_known_block_hash"] = peer->last_block_delegate_has_seen;
+        user_data["last_known_block_hash"] = fc::variant( peer->last_block_delegate_has_seen, 1 );
         user_data["last_known_block_number"] = _delegate->get_block_number(peer->last_block_delegate_has_seen);
         user_data["last_known_block_time"] = peer->last_block_time_delegate_has_seen;
 
@@ -4452,7 +4457,7 @@ namespace graphene { namespace net { namespace detail {
       {
         try
         {
-          _node_configuration = fc::json::from_file( configuration_file_name ).as<detail::node_configuration>();
+          _node_configuration = fc::json::from_file( configuration_file_name ).as<detail::node_configuration>(GRAPHENE_NET_MAX_NESTED_OBJECTS);
           ilog( "Loaded configuration from file ${filename}", ("filename", configuration_file_name ) );
 
           if( _node_configuration.private_key == fc::ecc::private_key() )
@@ -4816,20 +4821,19 @@ namespace graphene { namespace net { namespace detail {
         peer_to_disconnect->send_message( closing_message );
       }
 
-      // notify the user.  This will be useful in testing, but we might want to remove it later;
+      // notify the user.  This will be useful in testing, but we might want to remove it later.
-      // it makes good sense to notify the user if other nodes think she is behaving badly, but
+      // It makes good sense to notify the user if other nodes think she is behaving badly, but
       // if we're just detecting and dissconnecting other badly-behaving nodes, they don't really care.
       if (caused_by_error)
       {
         std::ostringstream error_message;
-        error_message << "I am disconnecting peer " << fc::variant( peer_to_disconnect->get_remote_endpoint() ).as_string() <<
+        error_message << "I am disconnecting peer " << fc::variant( peer_to_disconnect->get_remote_endpoint(), GRAPHENE_NET_MAX_NESTED_OBJECTS ).as_string() <<
                          " for reason: " << reason_for_disconnect;
         _delegate->error_encountered(error_message.str(), fc::oexception());
         dlog(error_message.str());
       }
       else
         dlog("Disconnecting from ${peer} for ${reason}", ("peer",peer_to_disconnect->get_remote_endpoint()) ("reason",reason_for_disconnect));
-      // peer_to_disconnect->close_connection();
     }
 
     void node_impl::listen_on_endpoint( const fc::ip::endpoint& ep, bool wait_if_not_available )
@@ -4888,7 +4892,7 @@ namespace graphene { namespace net { namespace detail {
         peer_details["version"] = "";
         peer_details["subver"] = peer->user_agent;
         peer_details["inbound"] = peer->direction == peer_connection_direction::inbound;
-        peer_details["firewall_status"] = peer->is_firewalled;
+        peer_details["firewall_status"] = fc::variant( peer->is_firewalled, 1 );
         peer_details["startingheight"] = "";
         peer_details["banscore"] = "";
         peer_details["syncnode"] = "";
@@ -4922,7 +4926,7 @@ namespace graphene { namespace net { namespace detail {
         // provide these for debugging
         // warning: these are just approximations, if the peer is "downstream" of us, they may
         // have received blocks from other peers that we are unaware of
-        peer_details["current_head_block"] = peer->last_block_delegate_has_seen;
+        peer_details["current_head_block"] = fc::variant( peer->last_block_delegate_has_seen, 1 );
         peer_details["current_head_block_number"] = _delegate->get_block_number(peer->last_block_delegate_has_seen);
         peer_details["current_head_block_time"] = peer->last_block_time_delegate_has_seen;
 
@@ -4998,17 +5002,17 @@ namespace graphene { namespace net { namespace detail {
     {
       VERIFY_CORRECT_THREAD();
       if (params.contains("peer_connection_retry_timeout"))
-        _peer_connection_retry_timeout = params["peer_connection_retry_timeout"].as<uint32_t>();
+        _peer_connection_retry_timeout = params["peer_connection_retry_timeout"].as<uint32_t>(1);
       if (params.contains("desired_number_of_connections"))
-        _desired_number_of_connections = params["desired_number_of_connections"].as<uint32_t>();
+        _desired_number_of_connections = params["desired_number_of_connections"].as<uint32_t>(1);
       if (params.contains("maximum_number_of_connections"))
-        _maximum_number_of_connections = params["maximum_number_of_connections"].as<uint32_t>();
+        _maximum_number_of_connections = params["maximum_number_of_connections"].as<uint32_t>(1);
       if (params.contains("maximum_number_of_blocks_to_handle_at_one_time"))
-        _maximum_number_of_blocks_to_handle_at_one_time = params["maximum_number_of_blocks_to_handle_at_one_time"].as<uint32_t>();
+        _maximum_number_of_blocks_to_handle_at_one_time = params["maximum_number_of_blocks_to_handle_at_one_time"].as<uint32_t>(1);
       if (params.contains("maximum_number_of_sync_blocks_to_prefetch"))
-        _maximum_number_of_sync_blocks_to_prefetch = params["maximum_number_of_sync_blocks_to_prefetch"].as<uint32_t>();
+        _maximum_number_of_sync_blocks_to_prefetch = params["maximum_number_of_sync_blocks_to_prefetch"].as<uint32_t>(1);
       if (params.contains("maximum_blocks_per_peer_during_syncing"))
-        _maximum_blocks_per_peer_during_syncing = params["maximum_blocks_per_peer_during_syncing"].as<uint32_t>();
+        _maximum_blocks_per_peer_during_syncing = params["maximum_blocks_per_peer_during_syncing"].as<uint32_t>(1);
 
       _desired_number_of_connections = std::min(_desired_number_of_connections, _maximum_number_of_connections);
 
@@ -5093,9 +5097,9 @@ namespace graphene { namespace net { namespace detail {
       VERIFY_CORRECT_THREAD();
       fc::mutable_variant_object info;
       info["listening_on"] = _actual_listening_endpoint;
-      info["node_public_key"] = _node_public_key;
+      info["node_public_key"] = fc::variant( _node_public_key, 1 );
-      info["node_id"] = _node_id;
+      info["node_id"] = fc::variant( _node_id, 1 );
-      info["firewalled"] = _is_firewalled;
+      info["firewalled"] = fc::variant( _is_firewalled, 1 );
       return info;
     }
     fc::variant_object node_impl::network_get_usage_stats() const
@@ -5123,9 +5127,9 @@ namespace graphene { namespace net { namespace detail {
                      std::plus<uint32_t>());
 
       fc::mutable_variant_object result;
-      result["usage_by_second"] = network_usage_by_second;
+      result["usage_by_second"] = fc::variant( network_usage_by_second, 2 );
-      result["usage_by_minute"] = network_usage_by_minute;
+      result["usage_by_minute"] = fc::variant( network_usage_by_minute, 2 );
-      result["usage_by_hour"] = network_usage_by_hour;
+      result["usage_by_hour"]   = fc::variant( network_usage_by_hour, 2 );
       return result;
     }
 

libraries/net/peer_database.cpp

@@ -34,8 +34,7 @@
 #include <fc/io/json.hpp>
 
 #include <graphene/net/peer_database.hpp>
-
+#include <graphene/net/config.hpp>
-
 
 namespace graphene { namespace net {
   namespace detail
@@ -81,7 +80,7 @@ namespace graphene { namespace net {
     public:
       typedef peer_database_impl::potential_peer_set::index<peer_database_impl::last_seen_time_index>::type::iterator last_seen_time_index_iterator;
       last_seen_time_index_iterator _iterator;
-      peer_database_iterator_impl(const last_seen_time_index_iterator& iterator) :
+      explicit peer_database_iterator_impl(const last_seen_time_index_iterator& iterator) :
         _iterator(iterator)
       {}
     };
@@ -95,9 +94,8 @@ namespace graphene { namespace net {
       {
         try
         {
-          std::vector<potential_peer_record> peer_records = fc::json::from_file(_peer_database_filename).as<std::vector<potential_peer_record> >();
+          std::vector<potential_peer_record> peer_records = fc::json::from_file(_peer_database_filename).as<std::vector<potential_peer_record> >( GRAPHENE_NET_MAX_NESTED_OBJECTS );
           std::copy(peer_records.begin(), peer_records.end(), std::inserter(_potential_peer_set, _potential_peer_set.end()));
-#define MAXIMUM_PEERDB_SIZE 1000
           if (_potential_peer_set.size() > MAXIMUM_PEERDB_SIZE)
           {
             // prune database to a reasonable size
@@ -125,7 +123,7 @@ namespace graphene { namespace net {
         fc::path peer_database_filename_dir = _peer_database_filename.parent_path();
         if (!fc::exists(peer_database_filename_dir))
           fc::create_directories(peer_database_filename_dir);
-        fc::json::save_to_file(peer_records, _peer_database_filename);
+        fc::json::save_to_file( peer_records, _peer_database_filename, GRAPHENE_NET_MAX_NESTED_OBJECTS );
       }
       catch (const fc::exception& e)
       {

libraries/plugins/account_history/account_history_plugin.cpp

@@ -81,11 +81,23 @@ void account_history_plugin_impl::update_account_histories( const signed_block&
 {
    graphene::chain::database& db = database();
    vector<optional< operation_history_object > >& hist = db.get_applied_operations();
+   bool is_first = true;
+   auto skip_oho_id = [&is_first,&db,this]() {
+      if( is_first && db._undo_db.enabled() ) // this ensures that the current id is rolled back on undo
+      {
+         db.remove( db.create<operation_history_object>( []( operation_history_object& obj) {} ) );
+         is_first = false;
+      }
+      else
+         _oho_index->use_next_id();
+   };
+
    for( optional< operation_history_object >& o_op : hist )
    {
       optional<operation_history_object> oho;
 
       auto create_oho = [&]() {
+         is_first = false;
          operation_history_object result = db.create<operation_history_object>( [&]( operation_history_object& h )
          {
             if( o_op.valid() )
@@ -99,7 +111,7 @@ void account_history_plugin_impl::update_account_histories( const signed_block&
       {
          // Note: the 2nd and 3rd checks above are for better performance, when the db is not clean,
          //       they will break consistency of account_stats.total_ops and removed_ops and most_recent_op
-         _oho_index->use_next_id();
+         skip_oho_id();
          continue;
       }
       else if( !_partial_operations )
@@ -117,7 +129,14 @@ void account_history_plugin_impl::update_account_histories( const signed_block&
          impacted.insert( op.result.get<object_id_type>() );
       else
          graphene::app::operation_get_impacted_accounts( op.op, impacted );
-
+      if( op.op.which() == operation::tag< lottery_end_operation >::value ) 
+      {
+         auto lop = op.op.get< lottery_end_operation >();
+         auto asset_object = lop.lottery( db );
+         impacted.insert( asset_object.issuer );
+         for( auto benefactor : asset_object.lottery_options->benefactors )
+            impacted.insert( benefactor.id );
+      }
       for( auto& a : other )
          for( auto& item : a.account_auths )
             impacted.insert( item.first );
@@ -172,7 +191,7 @@ void account_history_plugin_impl::update_account_histories( const signed_block&
          }
       }
       if (_partial_operations && ! oho.valid())
-         _oho_index->use_next_id();
+         skip_oho_id();
    }
 }
 

libraries/plugins/debug_witness/debug_api.cpp

@@ -22,12 +22,11 @@ namespace detail {
 class debug_api_impl
 {
    public:
-      debug_api_impl( graphene::app::application& _app );
+      explicit debug_api_impl( graphene::app::application& _app );
 
       void debug_push_blocks( const std::string& src_filename, uint32_t count );
       void debug_generate_blocks( const std::string& debug_key, uint32_t count );
       void debug_update_object( const fc::variant_object& update );
-      //void debug_save_db( std::string db_path );
       void debug_stream_json_objects( const std::string& filename );
       void debug_stream_json_objects_flush();
       std::shared_ptr< graphene::debug_witness_plugin::debug_witness_plugin > get_plugin();
@@ -71,7 +70,6 @@ void debug_api_impl::debug_push_blocks( const std::string& src_filename, uint32_
          }
       }
       ilog( "Completed loading block_database successfully" );
-      return;
    }
 }
 
@@ -93,7 +91,7 @@ void debug_api_impl::debug_generate_blocks( const std::string& debug_key, uint32
       if( scheduled_key != debug_public_key )
       {
          ilog( "Modified key for witness ${w}", ("w", scheduled_witness) );
-         fc::mutable_variant_object update;
+         fc::limited_mutable_variant_object update( GRAPHENE_MAX_NESTED_OBJECTS );
          update("_action", "update")("id", scheduled_witness)("signing_key", debug_public_key);
          db->debug_update( update );
       }

libraries/plugins/debug_witness/debug_witness.cpp

@@ -68,7 +68,7 @@ void debug_witness_plugin::plugin_initialize(const boost::program_options::varia
       const std::vector<std::string> key_id_to_wif_pair_strings = options["private-key"].as<std::vector<std::string>>();
       for (const std::string& key_id_to_wif_pair_string : key_id_to_wif_pair_strings)
       {
-         auto key_id_to_wif_pair = graphene::app::dejsonify<std::pair<chain::public_key_type, std::string> >(key_id_to_wif_pair_string);
+         auto key_id_to_wif_pair = graphene::app::dejsonify<std::pair<chain::public_key_type, std::string> >(key_id_to_wif_pair_string, GRAPHENE_MAX_NESTED_OBJECTS);
          idump((key_id_to_wif_pair));
          fc::optional<fc::ecc::private_key> private_key = graphene::utilities::wif_to_key(key_id_to_wif_pair.second);
          if (!private_key)
@@ -77,7 +77,7 @@ void debug_witness_plugin::plugin_initialize(const boost::program_options::varia
             // just here to ease the transition, can be removed soon
             try
             {
-               private_key = fc::variant(key_id_to_wif_pair.second).as<fc::ecc::private_key>();
+               private_key = fc::variant( key_id_to_wif_pair.second, GRAPHENE_MAX_NESTED_OBJECTS ).as<fc::ecc::private_key>( GRAPHENE_MAX_NESTED_OBJECTS );
             }
             catch (const fc::exception&)
             {

libraries/plugins/delayed_node/delayed_node_plugin.cpp

@@ -65,7 +65,7 @@ void delayed_node_plugin::plugin_set_program_options(bpo::options_description& c
 
 void delayed_node_plugin::connect()
 {
-   my->client_connection = std::make_shared<fc::rpc::websocket_api_connection>(*my->client.connect(my->remote_endpoint));
+   my->client_connection = std::make_shared<fc::rpc::websocket_api_connection>(my->client.connect(my->remote_endpoint), 1);
    my->database_api = my->client_connection->get_remote_api<graphene::app::database_api>(0);
    my->client_connection_closed = my->client_connection->closed.connect([this] {
       connection_failed();
@@ -143,7 +143,7 @@ void delayed_node_plugin::plugin_startup()
       connect();
       my->database_api->set_block_applied_callback([this]( const fc::variant& block_id )
       {
-         fc::from_variant( block_id, my->last_received_remote_head );
+         fc::from_variant( block_id, my->last_received_remote_head, GRAPHENE_MAX_NESTED_OBJECTS );
       } );
       return;
    }

libraries/plugins/generate_genesis/generate_genesis.cpp

@@ -113,7 +113,7 @@ std::string modify_account_name(const std::string& name)
 
 bool is_special_account(const graphene::chain::account_id_type& account_id)
 {
-    return account_id.instance < 100;
+    return account_id.instance.value < 100;
 }
 
     bool is_scam(const std::string& account_name)

libraries/plugins/generate_uia_sharedrop_genesis/generate_uia_sharedrop_genesis.cpp

@@ -122,7 +122,7 @@ namespace
 
     bool is_special_account(const graphene::chain::account_id_type& account_id)
     {
-        return account_id.instance < 100;
+        return account_id.instance.value < 100;
     }
 
     bool is_scam(const std::string& account_name)

libraries/plugins/grouped_orders/grouped_orders_plugin.cpp

@@ -0,0 +1,303 @@
+/*
+ * Copyright (c) 2018 Abit More, and contributors.
+ *
+ * The MIT License
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include <graphene/grouped_orders/grouped_orders_plugin.hpp>
+
+#include <graphene/chain/market_object.hpp>
+
+namespace graphene { namespace grouped_orders {
+
+namespace detail
+{
+
+class grouped_orders_plugin_impl
+{
+   public:
+      grouped_orders_plugin_impl(grouped_orders_plugin& _plugin)
+      :_self( _plugin ) {}
+      virtual ~grouped_orders_plugin_impl();
+
+      graphene::chain::database& database()
+      {
+         return _self.database();
+      }
+
+      grouped_orders_plugin&     _self;
+      flat_set<uint16_t>         _tracked_groups;
+};
+
+/**
+ *  @brief This secondary index is used to track changes on limit order objects.
+ */
+class limit_order_group_index : public secondary_index
+{
+   public:
+      limit_order_group_index( const flat_set<uint16_t>& groups ) : _tracked_groups( groups ) {};
+
+      virtual void object_inserted( const object& obj ) override;
+      virtual void object_removed( const object& obj ) override;
+      virtual void about_to_modify( const object& before ) override;
+      virtual void object_modified( const object& after  ) override;
+
+      const flat_set<uint16_t>& get_tracked_groups() const
+      { return _tracked_groups; }
+
+      const map< limit_order_group_key, limit_order_group_data >& get_order_groups() const
+      { return _og_data; }
+
+   private:
+      void remove_order( const limit_order_object& obj, bool remove_empty = true );
+
+      /** tracked groups */
+      flat_set<uint16_t> _tracked_groups;
+
+      /** maps the group key to group data */
+      map< limit_order_group_key, limit_order_group_data > _og_data;
+};
+
+void limit_order_group_index::object_inserted( const object& objct )
+{ try {
+   const limit_order_object& o = static_cast<const limit_order_object&>( objct );
+
+   auto& idx = _og_data;
+
+   for( uint16_t group : get_tracked_groups() )
+   {
+      auto create_ogo = [&]() {
+         idx[ limit_order_group_key( group, o.sell_price ) ] = limit_order_group_data( o.sell_price, o.for_sale );
+      };
+      // if idx is empty, insert this order
+      // Note: not capped
+      if( idx.empty() )
+      {
+         create_ogo();
+         continue;
+      }
+
+      // cap the price
+      price capped_price = o.sell_price;
+      price max = o.sell_price.max();
+      price min = o.sell_price.min();
+      bool capped_max = false;
+      bool capped_min = false;
+      if( o.sell_price > max )
+      {
+         capped_price = max;
+         capped_max = true;
+      }
+      else if( o.sell_price < min )
+      {
+         capped_price = min;
+         capped_min = true;
+      }
+      // if idx is not empty, find the group that is next to this order
+      auto itr = idx.lower_bound( limit_order_group_key( group, capped_price ) );
+      bool check_previous = false;
+      if( itr == idx.end() || itr->first.group != group
+            || itr->first.min_price.base.asset_id != o.sell_price.base.asset_id
+            || itr->first.min_price.quote.asset_id != o.sell_price.quote.asset_id )
+         // not same market or group type
+         check_previous = true;
+      else // same market and group type
+      {
+         bool update_max = false;
+         if( capped_price > itr->second.max_price ) // implies itr->min_price <= itr->max_price < max
+         {
+            update_max = true;
+            price max_price = itr->first.min_price * ratio_type( GRAPHENE_100_PERCENT + group, GRAPHENE_100_PERCENT );
+            // max_price should have been capped here
+            if( capped_price > max_price ) // new order is out of range
+               check_previous = true;
+         }
+         if( !check_previous ) // new order is within the range
+         {
+            if( capped_min && o.sell_price < itr->first.min_price )
+            {  // need to update itr->min_price here, if itr is below min, and new order is even lower
+               // TODO improve performance
+               limit_order_group_data data( itr->second.max_price, o.for_sale + itr->second.total_for_sale );
+               idx.erase( itr );
+               idx[ limit_order_group_key( group, o.sell_price ) ] = data;
+            }
+            else
+            {
+               if( update_max || ( capped_max && o.sell_price > itr->second.max_price ) )
+                  itr->second.max_price = o.sell_price; // store real price here, not capped
+               itr->second.total_for_sale += o.for_sale;
+            }
+         }
+      }
+
+      if( check_previous )
+      {
+         if( itr == idx.begin() ) // no previous
+            create_ogo();
+         else
+         {
+            --itr; // should be valid
+            if( itr->first.group != group || itr->first.min_price.base.asset_id != o.sell_price.base.asset_id
+                                          || itr->first.min_price.quote.asset_id != o.sell_price.quote.asset_id )
+               // not same market or group type
+               create_ogo();
+            else // same market and group type
+            {
+               // due to lower_bound, always true: capped_price < itr->first.min_price, so no need to check again,
+               // if new order is in range of itr group, always need to update itr->first.min_price, unless
+               //   o.sell_price is higher than max
+               price min_price = itr->second.max_price / ratio_type( GRAPHENE_100_PERCENT + group, GRAPHENE_100_PERCENT );
+               // min_price should have been capped here
+               if( capped_price < min_price ) // new order is out of range
+                  create_ogo();
+               else if( capped_max && o.sell_price >= itr->first.min_price )
+               {  // itr is above max, and price of new order is even higher
+                  if( o.sell_price > itr->second.max_price )
+                     itr->second.max_price = o.sell_price;
+                  itr->second.total_for_sale += o.for_sale;
+               }
+               else
+               {  // new order is within the range
+                  // TODO improve performance
+                  limit_order_group_data data( itr->second.max_price, o.for_sale + itr->second.total_for_sale );
+                  idx.erase( itr );
+                  idx[ limit_order_group_key( group, o.sell_price ) ] = data;
+               }
+            }
+         }
+      }
+   }
+} FC_CAPTURE_AND_RETHROW( (objct) ); }
+
+void limit_order_group_index::object_removed( const object& objct )
+{ try {
+   const limit_order_object& o = static_cast<const limit_order_object&>( objct );
+   remove_order( o );
+} FC_CAPTURE_AND_RETHROW( (objct) ); }
+
+void limit_order_group_index::about_to_modify( const object& objct )
+{ try {
+   const limit_order_object& o = static_cast<const limit_order_object&>( objct );
+   remove_order( o, false );
+} FC_CAPTURE_AND_RETHROW( (objct) ); }
+
+void limit_order_group_index::object_modified( const object& objct )
+{ try {
+   object_inserted( objct );
+} FC_CAPTURE_AND_RETHROW( (objct) ); }
+
+void limit_order_group_index::remove_order( const limit_order_object& o, bool remove_empty )
+{
+   auto& idx = _og_data;
+
+   for( uint16_t group : get_tracked_groups() )
+   {
+      // find the group that should contain this order
+      auto itr = idx.lower_bound( limit_order_group_key( group, o.sell_price ) );
+      if( itr == idx.end() || itr->first.group != group
+            || itr->first.min_price.base.asset_id != o.sell_price.base.asset_id
+            || itr->first.min_price.quote.asset_id != o.sell_price.quote.asset_id
+            || itr->second.max_price < o.sell_price )
+      {
+         // can not find corresponding group, should not happen
+         wlog( "can not find the order group containing order for removing (price dismatch): ${o}", ("o",o) );
+         continue;
+      }
+      else // found
+      {
+         if( itr->second.total_for_sale < o.for_sale )
+            // should not happen
+            wlog( "can not find the order group containing order for removing (amount dismatch): ${o}", ("o",o) );
+         else if( !remove_empty || itr->second.total_for_sale > o.for_sale )
+            itr->second.total_for_sale -= o.for_sale;
+         else
+            // it's the only order in the group and need to be removed
+            idx.erase( itr );
+      }
+   }
+}
+
+grouped_orders_plugin_impl::~grouped_orders_plugin_impl()
+{}
+
+} // end namespace detail
+
+
+grouped_orders_plugin::grouped_orders_plugin() :
+   my( new detail::grouped_orders_plugin_impl(*this) )
+{
+}
+
+grouped_orders_plugin::~grouped_orders_plugin()
+{
+}
+
+std::string grouped_orders_plugin::plugin_name()const
+{
+   return "grouped_orders";
+}
+
+void grouped_orders_plugin::plugin_set_program_options(
+   boost::program_options::options_description& cli,
+   boost::program_options::options_description& cfg
+   )
+{
+   cli.add_options()
+         ("tracked-groups", boost::program_options::value<string>()->default_value("[10,100]"), // 0.1% and 1%
+          "Group orders by percentage increase on price. Specify a JSON array of numbers here, each number is a group, number 1 means 0.01%. ")
+         ;
+   cfg.add(cli);
+}
+
+void grouped_orders_plugin::plugin_initialize(const boost::program_options::variables_map& options)
+{ try {
+
+   if( options.count( "tracked-groups" ) )
+   {
+      const std::string& groups = options["tracked-groups"].as<string>();
+      my->_tracked_groups = fc::json::from_string(groups).as<flat_set<uint16_t>>( 2 );
+      my->_tracked_groups.erase( 0 );
+   }
+   else
+      my->_tracked_groups = fc::json::from_string("[10,100]").as<flat_set<uint16_t>>(2);
+
+   database().add_secondary_index< primary_index<limit_order_index>, detail::limit_order_group_index >( my->_tracked_groups );
+
+} FC_CAPTURE_AND_RETHROW() }
+
+void grouped_orders_plugin::plugin_startup()
+{
+}
+
+const flat_set<uint16_t>& grouped_orders_plugin::tracked_groups() const
+{
+   return my->_tracked_groups;
+}
+
+const map< limit_order_group_key, limit_order_group_data >& grouped_orders_plugin::limit_order_groups()
+{
+   const auto& idx = database().get_index_type< limit_order_index >();
+   const auto& pidx = dynamic_cast<const primary_index< limit_order_index >&>(idx);
+   const auto& logidx = pidx.get_secondary_index< detail::limit_order_group_index >();
+   return logidx.get_order_groups();
+}
+
+} }