Why is it so slow when assigning a concatenated string to a variable in python?How to concatenate text from multiple rows into a single text string in SQL server?How do I parse a string to a float or int in Python?Python join: why is it string.join(list) instead of list.join(string)?How to substring a string in Python?Reverse a string in PythonConverting integer to string in Python?How do I concatenate two lists in Python?Does Python have a string 'contains' substring method?How to concatenate string variables in BashHow do I lowercase a string in Python?
Exception propagation: When to catch exceptions?
A musical commute to work
Is a vertical stabiliser needed for straight line flight in a glider?
How to make a language evolve quickly?
Can I use my laptop, which says 240V, in the USA?
Limit of an integral vs Limit of the integrand
Is Simic Ascendancy triggered by Awakening of Vitu-Ghazi?
Noob at soldering, can anyone explain why my circuit won't work?
Is a diamond sword feasible?
We are two immediate neighbors who forged our own powers to form concatenated relationship. Who are we?
How are one-time password generators like Google Authenticator different from having two passwords?
What can cause a never-frozen indoor copper drain pipe to crack?
Why does the Earth follow an elliptical trajectory rather than a parabolic one?
semanage not changing file context
find not returning expected files
Ex-manager wants to stay in touch, I don't want to
Why does a C.D.F need to be right-continuous?
Is the schwa sound consistent?
How to make the table in the figure in LaTeX?
Pre-1993 comic in which Wolverine's claws were turned to rubber?
Ubuntu won't let me edit or delete .vimrc file
Control variables and other independent variables
Why do Thanos's punches not kill Captain America or at least cause some mortal injuries?
Does Lawful Interception of 4G / the proposed 5G provide a back door for hackers as well?
Why is it so slow when assigning a concatenated string to a variable in python?
How to concatenate text from multiple rows into a single text string in SQL server?How do I parse a string to a float or int in Python?Python join: why is it string.join(list) instead of list.join(string)?How to substring a string in Python?Reverse a string in PythonConverting integer to string in Python?How do I concatenate two lists in Python?Does Python have a string 'contains' substring method?How to concatenate string variables in BashHow do I lowercase a string in Python?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
New contributor
add a comment |
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
New contributor
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then''.join
)
– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
add a comment |
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
New contributor
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
python string-concatenation
New contributor
New contributor
New contributor
asked 2 hours ago
uma66uma66
362
362
New contributor
New contributor
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then''.join
)
– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
add a comment |
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then''.join
)
– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
3
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then
''.join
)– juanpa.arrivillaga
2 hours ago
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then
''.join
)– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
add a comment |
1 Answer
1
active
oldest
votes
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f56086479%2fwhy-is-it-so-slow-when-assigning-a-concatenated-string-to-a-variable-in-python%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
edited 2 hours ago
answered 2 hours ago
ShadowRangerShadowRanger
65.4k664103
65.4k664103
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
This was exactly what I suspected.
– Tom Karzes
2 hours ago
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f56086479%2fwhy-is-it-so-slow-when-assigning-a-concatenated-string-to-a-variable-in-python%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then
''.join
)– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago