What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?
Disable all sound permanently
The answer is a girl's name (my future granddaughter) - can anyone help?
How dangerous is a very out-of-true disc brake wheel?
Re-entering the UK after overstaying in 2008
Manager told a colleague of mine I was getting fired soon
What's the global, general word that stands for "center tone of a song"?
Is there anything on the ISS that would be destroyed if that object were returned to Earth?
What does "execute a hard copy" mean?
Where does the image of a data connector as a sharp metal spike originate from?
What does the triangle look like in this diagram?
Is the "spacetime" the same thing as the mathematical 4th dimension?
How important is knowledge of trig identities for use in Calculus
Is there an in-universe explanation of how Frodo's arrival in Valinor was recorded in the Red Book?
What's the correct way to determine turn order in this situation?
How to level a picture frame hung on a single nail?
Realistically, how much do you need to start investing?
What is the score of my Scopa hand?
How to "Start as close to the end as possible", and why to do so?
Isn't the detector always measuring, and thus always collapsing the state?
What is the point of impeaching Trump?
Why such a singular place for bird watching?
Caro-Kann c4-c5 push
Does Bank Manager's discretion still exist in Mortgage Lending
Why do Russians sometimes spell "жирный" (fatty) as "жырный"?
What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?
INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
add a comment
|
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
2
Why not useINSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.
– ypercubeᵀᴹ
9 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
9 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
8 hours ago
add a comment
|
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
sql-server sql-server-2016 identity bulk-insert
edited 7 hours ago
Paul White♦
59.8k16 gold badges310 silver badges489 bronze badges
59.8k16 gold badges310 silver badges489 bronze badges
asked 9 hours ago
Daniel James BryarsDaniel James Bryars
3791 gold badge3 silver badges16 bronze badges
3791 gold badge3 silver badges16 bronze badges
2
Why not useINSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.
– ypercubeᵀᴹ
9 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
9 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
8 hours ago
add a comment
|
2
Why not useINSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.
– ypercubeᵀᴹ
9 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
9 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
8 hours ago
2
2
Why not use
INSERT .. SELECT ..
with OUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.– ypercubeᵀᴹ
9 hours ago
Why not use
INSERT .. SELECT ..
with OUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.– ypercubeᵀᴹ
9 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
9 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
9 hours ago
2
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
8 hours ago
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
8 hours ago
add a comment
|
1 Answer
1
active
oldest
votes
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
add a comment
|
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "182"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
add a comment
|
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
add a comment
|
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
edited 7 hours ago
answered 8 hours ago
PiotrPiotr
4089 bronze badges
4089 bronze badges
add a comment
|
add a comment
|
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Why not use
INSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.– ypercubeᵀᴹ
9 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
9 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
8 hours ago