2,104 questions
Best practices
1
vote
3
replies
74
views
Deleting large data without stopping the active mysql server
I'm using a table of approximately 1TB in a MySQL database. This table also has a monthly partition. We store the last two months of data in this table and regularly truncate the data from the ...
- 21
Best practices
0
votes
3
replies
49
views
I want to make "HTTPS proxy cache server" with nginx that controls large size of git source
i'm newbie in nginx and proxy server.
We have a problem about googlesource 429 Error, caused by many requests
and because of bandwidth, we took a long time to get googlesource.
We reviewed to make ...
0
votes
1
answer
83
views
How to reference a second Pandas dataframe to the first one without creating any copy of the first one?
I have a large pandas dataframe df of something like a million rows and 100 columns, and I have to create a second dataframe df_n, same size as the first one. Several rows and columns of df_n will be ...
- 165
1
vote
1
answer
103
views
pandas read csv with MANY columns
I have a csv that has 162 rows, but around 3 million columns. I want to read it with pandas. I have enough RAM available, but pd.read_csv(file.csv, header=None, dtype=str) takes forever. The cells ...
- 25
0
votes
1
answer
120
views
Mat-Autocomplete with cdk-virtual-scroll-viewport (large data) arrow-keys didnt work
I am trying to use mat-autocomplete with large data. For the large data aspect im using a cdk-virtual-scroll-viewport with mat-autocomplete. Everything works except the Arrow-Key Navigation.
<mat-...
- 3
2
votes
0
answers
74
views
How can I optimize a PHP script that fetches 1M+ MySQL rows for reporting without exhausting memory? [duplicate]
I'm working on a PHP web application that generates reports from a MySQL table with over 1 million rows.
What I'm trying to do:
Fetch a large dataset and process it to generate a downloadable report (...
6
votes
3
answers
284
views
How to select element from two complex number lists to get minimum magnitude for their sum
I have two python lists(list1 and list2),each containing 51 complex numbers. At each index i, I can choose either list1[i] or list2[i]. I want to select one element per index(Total of 51 elements) ...
- 75
1
vote
1
answer
51
views
Date Range Large Index/Match Duplicate
The results area is finding the largest top 4 costs in column A within the date range =IFERROR(LARGE(IF(Sheet1!$D$5:$D$4935>=$A$2,IF(Sheet1!$D$5:$D$20<=$B$2,Sheet1!$E$5:$E$20)),1),0)and then ...
- 221
0
votes
1
answer
81
views
work with large matrices to build phylogenetic trees
Following this guide, I managed to convert my IBS matrix into a phylo object. Now. everything works just fine for small test cases.
However, I'm now trying to scale to the entire dataset of 300 ...
- 435
5
votes
1
answer
244
views
LOESS on very large dataset
I'm working with a very large dataset containing CWD (Cumulative Water Deficit) and EVI (Enhanced Vegetation Index) measurements across different landcover types. The current code uses LOESS ...
- 339
0
votes
1
answer
55
views
What is the problem when I am trying to remove "Don't know/refuse" level in SAS
I am trying to remove the "Don't know/refuse" for the headache and breasttenderness variable but all the values for the breasttenderness_num and headache_num are missing. Below is the code ...
- 63
-3
votes
2
answers
82
views
Identify data groups in irregular row groups in a dataframe in R
I want to know how many groups of data this df has:
df <- data.frame(
stringsAsFactors = FALSE,
V1 = c("A","-","-","-","B"...
- 448
0
votes
0
answers
55
views
How can I optimize and apply this same logic for large datasets?
I’m working on a system where I calculate the similarity between user vectors and product vectors using cosine similarity in Python with NumPy. The code below performs the necessary operations, but I ...
- 1
1
vote
0
answers
80
views
Larger-than-memory Survey Analysis with R+Arrow
I'm currently trying to analyze data from the National Inpatient Sample (NIS). When combining multiple years worth of data, my files are just over 8 GB after processing/selecting relevant columns. I ...
- 337
1
vote
1
answer
74
views
How to improve responsiveness of interactive plotly generated line plots saved as html files?
I have some very long time series data (millions of data points) and generate interactive plotly html plots based on this data. I am using Scattergl from plotly's graphical_obects.
When I attempt to ...
- 1,172