What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?

Disable all sound permanently

The answer is a girl's name (my future granddaughter) - can anyone help?

How dangerous is a very out-of-true disc brake wheel?

Re-entering the UK after overstaying in 2008

Manager told a colleague of mine I was getting fired soon

What's the global, general word that stands for "center tone of a song"?

Is there anything on the ISS that would be destroyed if that object were returned to Earth?

What does "execute a hard copy" mean?

Where does the image of a data connector as a sharp metal spike originate from?

What does the triangle look like in this diagram?

Is the "spacetime" the same thing as the mathematical 4th dimension?

How important is knowledge of trig identities for use in Calculus

Is there an in-universe explanation of how Frodo's arrival in Valinor was recorded in the Red Book?

What's the correct way to determine turn order in this situation?

How to level a picture frame hung on a single nail?

Realistically, how much do you need to start investing?

What is the score of my Scopa hand?

How to "Start as close to the end as possible", and why to do so?

Isn't the detector always measuring, and thus always collapsing the state?

What is the point of impeaching Trump?

Why such a singular place for bird watching?

Caro-Kann c4-c5 push

Does Bank Manager's discretion still exist in Mortgage Lending

Why do Russians sometimes spell "жирный" (fatty) as "жырный"?



What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?


INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;









4















I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.










share|improve this question





















  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    9 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    9 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    8 hours ago


















4















I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.










share|improve this question





















  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    9 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    9 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    8 hours ago














4












4








4








I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.










share|improve this question
















I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.







sql-server sql-server-2016 identity bulk-insert






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 7 hours ago









Paul White

59.8k16 gold badges310 silver badges489 bronze badges




59.8k16 gold badges310 silver badges489 bronze badges










asked 9 hours ago









Daniel James BryarsDaniel James Bryars

3791 gold badge3 silver badges16 bronze badges




3791 gold badge3 silver badges16 bronze badges










  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    9 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    9 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    8 hours ago













  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    9 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    9 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    8 hours ago








2




2





Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

– ypercubeᵀᴹ
9 hours ago






Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

– ypercubeᵀᴹ
9 hours ago














@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

– Daniel James Bryars
9 hours ago





@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

– Daniel James Bryars
9 hours ago




2




2





What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

– ypercubeᵀᴹ
8 hours ago






What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

– ypercubeᵀᴹ
8 hours ago











1 Answer
1






active

oldest

votes


















6
















You can use procedure (introduced in SQL Server 2012):
sp_sequence_get_range



To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



There is an example:



CREATE SCHEMA Test ; 
GO

CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;

CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;

DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;

EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;


Documentation: sp_sequence_get_range






share|improve this answer




























    Your Answer








    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "182"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );














    draft saved

    draft discarded
















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    6
















    You can use procedure (introduced in SQL Server 2012):
    sp_sequence_get_range



    To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



    There is an example:



    CREATE SCHEMA Test ; 
    GO

    CREATE SEQUENCE Test.RangeSeq
    AS int
    START WITH 1
    INCREMENT BY 1
    CACHE 10
    ;

    CREATE TABLE Test.ProcessEvents
    (
    EventID int PRIMARY KEY CLUSTERED
    DEFAULT (NEXT VALUE FOR Test.RangeSeq),
    EventTime datetime NOT NULL DEFAULT (getdate()),
    EventCode nvarchar(5) NOT NULL,
    Description nvarchar(300) NULL
    ) ;

    DECLARE @range_first_value sql_variant ,
    @range_first_value_output sql_variant ;

    EXEC sp_sequence_get_range
    @sequence_name = N'Test.RangeSeq'
    , @range_size = 4
    , @range_first_value = @range_first_value_output OUTPUT ;


    Documentation: sp_sequence_get_range






    share|improve this answer































      6
















      You can use procedure (introduced in SQL Server 2012):
      sp_sequence_get_range



      To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



      There is an example:



      CREATE SCHEMA Test ; 
      GO

      CREATE SEQUENCE Test.RangeSeq
      AS int
      START WITH 1
      INCREMENT BY 1
      CACHE 10
      ;

      CREATE TABLE Test.ProcessEvents
      (
      EventID int PRIMARY KEY CLUSTERED
      DEFAULT (NEXT VALUE FOR Test.RangeSeq),
      EventTime datetime NOT NULL DEFAULT (getdate()),
      EventCode nvarchar(5) NOT NULL,
      Description nvarchar(300) NULL
      ) ;

      DECLARE @range_first_value sql_variant ,
      @range_first_value_output sql_variant ;

      EXEC sp_sequence_get_range
      @sequence_name = N'Test.RangeSeq'
      , @range_size = 4
      , @range_first_value = @range_first_value_output OUTPUT ;


      Documentation: sp_sequence_get_range






      share|improve this answer





























        6














        6










        6









        You can use procedure (introduced in SQL Server 2012):
        sp_sequence_get_range



        To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



        There is an example:



        CREATE SCHEMA Test ; 
        GO

        CREATE SEQUENCE Test.RangeSeq
        AS int
        START WITH 1
        INCREMENT BY 1
        CACHE 10
        ;

        CREATE TABLE Test.ProcessEvents
        (
        EventID int PRIMARY KEY CLUSTERED
        DEFAULT (NEXT VALUE FOR Test.RangeSeq),
        EventTime datetime NOT NULL DEFAULT (getdate()),
        EventCode nvarchar(5) NOT NULL,
        Description nvarchar(300) NULL
        ) ;

        DECLARE @range_first_value sql_variant ,
        @range_first_value_output sql_variant ;

        EXEC sp_sequence_get_range
        @sequence_name = N'Test.RangeSeq'
        , @range_size = 4
        , @range_first_value = @range_first_value_output OUTPUT ;


        Documentation: sp_sequence_get_range






        share|improve this answer















        You can use procedure (introduced in SQL Server 2012):
        sp_sequence_get_range



        To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



        There is an example:



        CREATE SCHEMA Test ; 
        GO

        CREATE SEQUENCE Test.RangeSeq
        AS int
        START WITH 1
        INCREMENT BY 1
        CACHE 10
        ;

        CREATE TABLE Test.ProcessEvents
        (
        EventID int PRIMARY KEY CLUSTERED
        DEFAULT (NEXT VALUE FOR Test.RangeSeq),
        EventTime datetime NOT NULL DEFAULT (getdate()),
        EventCode nvarchar(5) NOT NULL,
        Description nvarchar(300) NULL
        ) ;

        DECLARE @range_first_value sql_variant ,
        @range_first_value_output sql_variant ;

        EXEC sp_sequence_get_range
        @sequence_name = N'Test.RangeSeq'
        , @range_size = 4
        , @range_first_value = @range_first_value_output OUTPUT ;


        Documentation: sp_sequence_get_range







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited 7 hours ago

























        answered 8 hours ago









        PiotrPiotr

        4089 bronze badges




        4089 bronze badges































            draft saved

            draft discarded















































            Thanks for contributing an answer to Database Administrators Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Invision Community Contents History See also References External links Navigation menuProprietaryinvisioncommunity.comIPS Community ForumsIPS Community Forumsthis blog entry"License Changes, IP.Board 3.4, and the Future""Interview -- Matt Mecham of Ibforums""CEO Invision Power Board, Matt Mecham Is a Liar, Thief!"IPB License Explanation 1.3, 1.3.1, 2.0, and 2.1ArchivedSecurity Fixes, Updates And Enhancements For IPB 1.3.1Archived"New Demo Accounts - Invision Power Services"the original"New Default Skin"the original"Invision Power Board 3.0.0 and Applications Released"the original"Archived copy"the original"Perpetual licenses being done away with""Release Notes - Invision Power Services""Introducing: IPS Community Suite 4!"Invision Community Release Notes

            Canceling a color specificationRandomly assigning color to Graphics3D objects?Default color for Filling in Mathematica 9Coloring specific elements of sets with a prime modified order in an array plotHow to pick a color differing significantly from the colors already in a given color list?Detection of the text colorColor numbers based on their valueCan color schemes for use with ColorData include opacity specification?My dynamic color schemes

            Ласкавець круглолистий Зміст Опис | Поширення | Галерея | Примітки | Посилання | Навігаційне меню58171138361-22960890446Bupleurum rotundifoliumEuro+Med PlantbasePlants of the World Online — Kew ScienceGermplasm Resources Information Network (GRIN)Ласкавецькн. VI : Літери Ком — Левиправивши або дописавши її